[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:02:29 dist-test-slave-wgsb krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:02:29 dist-test-slave-wgsb krb5kdc[399](info): set up 2 sockets
Sep 24 03:02:29 dist-test-slave-wgsb krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 03:02:30.812150 392 mini_kdc.cc:121] Time spent starting KDC: real 2.506s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:02:30 dist-test-slave-wgsb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:02:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:02:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:02:30.940014 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44417
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44417
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:31.024294 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:31.024528 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:31.024550 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:31.025743 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:02:31.025789 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:31.025803 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:31.025816 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:02:31.025830 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:02:31.027668 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:31.027926 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:31.028193 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:31.033569 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:31.033582 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:31.033595 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:31.033627 412 server_base.cc:1034] running on GCE node
I20260924 03:02:31.034031 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:31.034374 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:31.035562 412 hybrid_clock.cc:648] HybridClock initialized: now 1790218951035552 us; error 34 us; skew 500 ppm
Sep 24 03:02:31 dist-test-slave-wgsb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:31.038277 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:02:31.038810 412 webserver.cc:460] Webserver started at http://127.0.98.62:38347/ using document root <none> and password file <none>
I20260924 03:02:31.039064 412 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:31.039134 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:31.039291 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:31.040515 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "ba0f06c6452e4484a4cd64d928a01fa9"
format_stamp: "Formatted at 2026-09-24 03:02:31 on dist-test-slave-wgsb"
server_key: "6a7271b82bd1ad42f3a637841276cb76"
server_key_iv: "e2dfe9bf259c15e6fa4fbdc2af6f67fa"
server_key_version: "encryptionkey@0"
I20260924 03:02:31.040915 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "ba0f06c6452e4484a4cd64d928a01fa9"
format_stamp: "Formatted at 2026-09-24 03:02:31 on dist-test-slave-wgsb"
server_key: "6a7271b82bd1ad42f3a637841276cb76"
server_key_iv: "e2dfe9bf259c15e6fa4fbdc2af6f67fa"
server_key_version: "encryptionkey@0"
I20260924 03:02:31.042410 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:02:31.043255 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:31.043462 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:31.043578 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "ba0f06c6452e4484a4cd64d928a01fa9"
format_stamp: "Formatted at 2026-09-24 03:02:31 on dist-test-slave-wgsb"
server_key: "6a7271b82bd1ad42f3a637841276cb76"
server_key_iv: "e2dfe9bf259c15e6fa4fbdc2af6f67fa"
server_key_version: "encryptionkey@0"
I20260924 03:02:31.043725 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:31.069245 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:31.071425 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:31.071635 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:31.075949 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44417
I20260924 03:02:31.076028 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44417 every 8 connection(s)
I20260924 03:02:31.076397 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:02:31.077351 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:31.079959 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba0f06c6452e4484a4cd64d928a01fa9: Bootstrap starting.
I20260924 03:02:31.080590 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba0f06c6452e4484a4cd64d928a01fa9: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:31.080924 480 log.cc:824] T 00000000000000000000000000000000 P ba0f06c6452e4484a4cd64d928a01fa9: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:31.081707 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba0f06c6452e4484a4cd64d928a01fa9: No bootstrap required, opened a new log
I20260924 03:02:31.083436 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba0f06c6452e4484a4cd64d928a01fa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0f06c6452e4484a4cd64d928a01fa9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44417 } }
I20260924 03:02:31.083639 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba0f06c6452e4484a4cd64d928a01fa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:31.083711 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba0f06c6452e4484a4cd64d928a01fa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0f06c6452e4484a4cd64d928a01fa9, State: Initialized, Role: FOLLOWER
I20260924 03:02:31.083866 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba0f06c6452e4484a4cd64d928a01fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0f06c6452e4484a4cd64d928a01fa9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44417 } }
I20260924 03:02:31.083936 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba0f06c6452e4484a4cd64d928a01fa9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:02:31.084003 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba0f06c6452e4484a4cd64d928a01fa9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:02:31.084057 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba0f06c6452e4484a4cd64d928a01fa9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:31.084633 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba0f06c6452e4484a4cd64d928a01fa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0f06c6452e4484a4cd64d928a01fa9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44417 } }
I20260924 03:02:31.084750 480 leader_election.cc:302] T 00000000000000000000000000000000 P ba0f06c6452e4484a4cd64d928a01fa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba0f06c6452e4484a4cd64d928a01fa9; no voters:
I20260924 03:02:31.084954 480 leader_election.cc:288] T 00000000000000000000000000000000 P ba0f06c6452e4484a4cd64d928a01fa9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:02:31.084988 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba0f06c6452e4484a4cd64d928a01fa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:31.085143 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba0f06c6452e4484a4cd64d928a01fa9 [term 1 LEADER]: Becoming Leader. State: Replica: ba0f06c6452e4484a4cd64d928a01fa9, State: Running, Role: LEADER
I20260924 03:02:31.085237 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba0f06c6452e4484a4cd64d928a01fa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0f06c6452e4484a4cd64d928a01fa9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44417 } }
I20260924 03:02:31.085600 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 412
I20260924 03:02:31.085609 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba0f06c6452e4484a4cd64d928a01fa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba0f06c6452e4484a4cd64d928a01fa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0f06c6452e4484a4cd64d928a01fa9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44417 } } }
I20260924 03:02:31.085695 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba0f06c6452e4484a4cd64d928a01fa9 [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:31.085718 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:02:31.085778 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba0f06c6452e4484a4cd64d928a01fa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba0f06c6452e4484a4cd64d928a01fa9. Latest consensus state: current_term: 1 leader_uuid: "ba0f06c6452e4484a4cd64d928a01fa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0f06c6452e4484a4cd64d928a01fa9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44417 } } }
I20260924 03:02:31.085842 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba0f06c6452e4484a4cd64d928a01fa9 [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:31.086470 392 external_mini_cluster.cc:1428] Setting key 40585b9201fb8768d98c1dae385ce15c
I20260924 03:02:31.087024 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba0f06c6452e4484a4cd64d928a01fa9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:02:31.087092 487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:02:31.087579 487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:02:31.089138 487 catalog_manager.cc:1347] Generated new cluster ID: f1bac8db87154e3ba0be343796029589
I20260924 03:02:31.089192 487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:02:31.099277 487 catalog_manager.cc:1370] Generated new certificate authority record
Sep 24 03:02:31 dist-test-slave-wgsb krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:31.100033 487 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:02:31.103909 489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:31.087264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44906 (local address 127.0.98.62:44417)
0924 03:02:31.087751 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:31.087762 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:31.088853 (+ 1091us) server_negotiation.cc:407] Connection header received
0924 03:02:31.090155 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:31.090162 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:31.090219 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:31.090372 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:31.092366 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:31.093759 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:31.095154 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:31.095899 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:31.096043 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:31.100218 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:31.100222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:31.100223 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:31.100241 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:31.101971 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:31.102430 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:31.102433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:31.102434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:31.102523 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:31.102810 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:31.102811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:31.102811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:31.103026 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:31.103191 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:31.103466 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 03:02:31.103636 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":92,"threads_started":1}
I20260924 03:02:31.105916 487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba0f06c6452e4484a4cd64d928a01fa9: Generated new TSK 0
I20260924 03:02:31.106089 487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:02:31.138572 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44417
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:31.223541 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:31.223865 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:31.223894 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:31.225278 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:31.225329 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:31.225363 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:02:31.227065 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44417
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:31.227344 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:31.227727 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:31.234519 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:31.234462 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:31.234462 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:31.234720 506 server_base.cc:1034] running on GCE node
I20260924 03:02:31.234963 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:31.235232 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:31.236393 506 hybrid_clock.cc:648] HybridClock initialized: now 1790218951236379 us; error 32 us; skew 500 ppm
Sep 24 03:02:31 dist-test-slave-wgsb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:31.239342 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:02:31.239897 506 webserver.cc:460] Webserver started at http://127.0.98.1:38167/ using document root <none> and password file <none>
I20260924 03:02:31.240147 506 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:31.240216 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:31.240366 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:31.241827 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-0/data/instance:
uuid: "5799c675a59442ad827327056d00bcd7"
format_stamp: "Formatted at 2026-09-24 03:02:31 on dist-test-slave-wgsb"
server_key: "82d93c72f342eb03330bab02af0e2825"
server_key_iv: "10fe9641ed120e9852fd83379a4e6514"
server_key_version: "encryptionkey@0"
I20260924 03:02:31.242287 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5799c675a59442ad827327056d00bcd7"
format_stamp: "Formatted at 2026-09-24 03:02:31 on dist-test-slave-wgsb"
server_key: "82d93c72f342eb03330bab02af0e2825"
server_key_iv: "10fe9641ed120e9852fd83379a4e6514"
server_key_version: "encryptionkey@0"
I20260924 03:02:31.244005 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260924 03:02:31.244891 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:31.245193 506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:31.245307 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "5799c675a59442ad827327056d00bcd7"
format_stamp: "Formatted at 2026-09-24 03:02:31 on dist-test-slave-wgsb"
server_key: "82d93c72f342eb03330bab02af0e2825"
server_key_iv: "10fe9641ed120e9852fd83379a4e6514"
server_key_version: "encryptionkey@0"
I20260924 03:02:31.245386 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:31.269306 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:31.272279 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:31.272495 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:31.272876 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:31.273263 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:31.273329 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:31.273368 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:31.273401 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:31.279917 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46031
I20260924 03:02:31.279974 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46031 every 8 connection(s)
I20260924 03:02:31.280376 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:02:31.283819 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 506
I20260924 03:02:31.283943 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:02:31.284111 392 external_mini_cluster.cc:1428] Setting key a8f31658d968c129192181288524020f
Sep 24 03:02:31 dist-test-slave-wgsb krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:31.292407 489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:31.281224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36321 (local address 127.0.98.62:44417)
0924 03:02:31.281318 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:31.281320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:31.281516 (+ 196us) server_negotiation.cc:407] Connection header received
0924 03:02:31.282085 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:31.282086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:31.282097 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:31.282163 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:31.283281 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:31.283814 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:31.285591 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:31.285794 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:31.285859 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:31.288949 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:31.288950 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:31.288950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:31.288957 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:31.290507 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:31.291128 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:31.291130 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:31.291130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:31.291188 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:31.291562 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:31.291563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:31.291564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:31.291784 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:31.291849 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:31.292253 (+ 404us) server_negotiation.cc:299] Negotiation successful
0924 03:02:31.292341 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:02:31.292712 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:31.281179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44417 (local address 127.0.98.1:36321)
0924 03:02:31.281381 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:31.281429 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:31.281947 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:31.282257 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:31.282261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:31.282927 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:31.283172 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:31.283178 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:31.283913 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:31.283915 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:31.285310 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:31.285313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:31.285893 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:31.285894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:31.286061 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:31.286848 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:31.286858 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:31.288830 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:31.290602 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:31.290609 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:31.290612 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:31.290997 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:31.291263 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:31.291264 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:31.291265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:31.291439 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:31.291867 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:31.291875 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:31.292084 (+ 209us) client_negotiation.cc:769] Sending connection context
0924 03:02:31.292233 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 03:02:31.292429 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":102,"threads_started":1}
I20260924 03:02:31.293334 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44417
I20260924 03:02:31.293438 635 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:31.293646 635 heartbeater.cc:511] Master 127.0.98.62:44417 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 03:02:31.294269 444 ts_manager.cc:194] Registered new tserver with Master: 5799c675a59442ad827327056d00bcd7 (127.0.98.1:46031)
I20260924 03:02:31.294951 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:02:31.315572 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44417
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:31.397428 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:31.397694 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:31.397739 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:31.398962 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:31.399039 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:31.399083 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:02:31.400707 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44417
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:31.400996 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:31.401243 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:31.406517 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:31.406499 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:31.406524 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:31.406548 640 server_base.cc:1034] running on GCE node
I20260924 03:02:31.406935 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:31.407218 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:31.408384 640 hybrid_clock.cc:648] HybridClock initialized: now 1790218951408368 us; error 27 us; skew 500 ppm
Sep 24 03:02:31 dist-test-slave-wgsb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:31.411060 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:02:31.411626 640 webserver.cc:460] Webserver started at http://127.0.98.2:38691/ using document root <none> and password file <none>
I20260924 03:02:31.411888 640 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:31.411955 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:31.412129 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:31.413187 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-1/data/instance:
uuid: "a740b4f2a9494b9bbcaf1153154d5ff0"
format_stamp: "Formatted at 2026-09-24 03:02:31 on dist-test-slave-wgsb"
server_key: "f53e83317259b3e6a7766817b83752e3"
server_key_iv: "9523a611952abd27d0880ac665161e13"
server_key_version: "encryptionkey@0"
I20260924 03:02:31.413570 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a740b4f2a9494b9bbcaf1153154d5ff0"
format_stamp: "Formatted at 2026-09-24 03:02:31 on dist-test-slave-wgsb"
server_key: "f53e83317259b3e6a7766817b83752e3"
server_key_iv: "9523a611952abd27d0880ac665161e13"
server_key_version: "encryptionkey@0"
I20260924 03:02:31.414978 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:02:31.415927 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:31.416103 640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:31.416229 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "a740b4f2a9494b9bbcaf1153154d5ff0"
format_stamp: "Formatted at 2026-09-24 03:02:31 on dist-test-slave-wgsb"
server_key: "f53e83317259b3e6a7766817b83752e3"
server_key_iv: "9523a611952abd27d0880ac665161e13"
server_key_version: "encryptionkey@0"
I20260924 03:02:31.416304 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:31.433990 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:31.436645 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:31.436836 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:31.437106 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:31.437453 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:31.437520 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:31.437568 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:31.437595 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:31.444937 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40597
I20260924 03:02:31.444988 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40597 every 8 connection(s)
I20260924 03:02:31.445402 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:02:31.450812 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 640
I20260924 03:02:31.450932 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:02:31.451124 392 external_mini_cluster.cc:1428] Setting key df14a91b587399cc8d5c423d921d78c9
Sep 24 03:02:31 dist-test-slave-wgsb krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:31.456979 489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:31.446167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52957 (local address 127.0.98.62:44417)
0924 03:02:31.446280 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:31.446283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:31.446643 (+ 360us) server_negotiation.cc:407] Connection header received
0924 03:02:31.447225 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:31.447226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:31.447238 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:31.447291 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:31.448538 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:31.449145 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:31.450276 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:31.450477 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:31.450537 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:31.453300 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:31.453302 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:31.453302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:31.453312 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:31.454889 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:31.455395 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:31.455397 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:31.455397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:31.455957 (+ 560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:31.456251 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:31.456252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:31.456252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:31.456389 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:31.456470 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:31.456775 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 03:02:31.456866 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 03:02:31.457202 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:31.446150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44417 (local address 127.0.98.2:52957)
0924 03:02:31.446463 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:31.446517 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:31.447099 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:31.447394 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:31.447397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:31.448145 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:31.448353 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:31.448358 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:31.449270 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:31.449271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:31.450192 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:31.450195 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:31.450558 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:31.450559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:31.450676 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:31.451305 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:31.451316 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:31.453185 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:31.455027 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:31.455035 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:31.455037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:31.455292 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:31.456039 (+ 747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:31.456040 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:31.456040 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:31.456175 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:31.456460 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:31.456476 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:31.456631 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 03:02:31.456752 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 03:02:31.456924 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":117,"threads_started":1}
I20260924 03:02:31.457932 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44417
I20260924 03:02:31.458055 769 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:31.458238 769 heartbeater.cc:511] Master 127.0.98.62:44417 requested a full tablet report, sending...
I20260924 03:02:31.458745 444 ts_manager.cc:194] Registered new tserver with Master: a740b4f2a9494b9bbcaf1153154d5ff0 (127.0.98.2:40597)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 03:02:31.459327 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:02:31.479826 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44417
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:31.566102 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:31.566327 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:31.566352 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:31.567660 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:31.567714 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:31.567744 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:02:31.569264 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44417
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:31.569541 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:31.569792 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:31.575264 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:31.575264 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:31.575268 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:31.575557 774 server_base.cc:1034] running on GCE node
I20260924 03:02:31.575774 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:31.576048 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:31.577230 774 hybrid_clock.cc:648] HybridClock initialized: now 1790218951577217 us; error 47 us; skew 500 ppm
Sep 24 03:02:31 dist-test-slave-wgsb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:31.579995 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:02:31.580459 774 webserver.cc:460] Webserver started at http://127.0.98.3:34759/ using document root <none> and password file <none>
I20260924 03:02:31.580710 774 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:31.580785 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:31.580933 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:31.582106 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-2/data/instance:
uuid: "9e8f50538c124032920dc01aaa34853d"
format_stamp: "Formatted at 2026-09-24 03:02:31 on dist-test-slave-wgsb"
server_key: "73ad01eaa0be23bd63fb910c4d5b7851"
server_key_iv: "e04bc118089461f45e863dba0f0410be"
server_key_version: "encryptionkey@0"
I20260924 03:02:31.582521 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9e8f50538c124032920dc01aaa34853d"
format_stamp: "Formatted at 2026-09-24 03:02:31 on dist-test-slave-wgsb"
server_key: "73ad01eaa0be23bd63fb910c4d5b7851"
server_key_iv: "e04bc118089461f45e863dba0f0410be"
server_key_version: "encryptionkey@0"
I20260924 03:02:31.583969 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:02:31.584932 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:31.585098 774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:31.585198 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "9e8f50538c124032920dc01aaa34853d"
format_stamp: "Formatted at 2026-09-24 03:02:31 on dist-test-slave-wgsb"
server_key: "73ad01eaa0be23bd63fb910c4d5b7851"
server_key_iv: "e04bc118089461f45e863dba0f0410be"
server_key_version: "encryptionkey@0"
I20260924 03:02:31.585281 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:31.606371 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:31.608573 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:31.608742 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:31.609371 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:31.609695 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:31.609750 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:31.609794 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:31.609818 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:31.616995 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39243
I20260924 03:02:31.617131 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39243 every 8 connection(s)
I20260924 03:02:31.617431 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 03:02:31 dist-test-slave-wgsb krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:31.625358 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 774
I20260924 03:02:31.625427 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218948286999-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:02:31.625588 392 external_mini_cluster.cc:1428] Setting key 59872bc08a94099749d1bb266771527b
I20260924 03:02:31.627187 489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:31.618164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45655 (local address 127.0.98.62:44417)
0924 03:02:31.618231 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:31.618233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:31.618474 (+ 241us) server_negotiation.cc:407] Connection header received
0924 03:02:31.619003 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:31.619005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:31.619049 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:31.619105 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:31.620217 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:31.620716 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:31.621751 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:31.621961 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:31.622051 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:31.624264 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:31.624266 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:31.624266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:31.624273 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:31.625733 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:31.626201 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:31.626202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:31.626203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:31.626257 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:31.626510 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:31.626511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:31.626511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:31.626640 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:31.626720 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:31.626992 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 03:02:31.627100 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 03:02:31.627377 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:31.618168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44417 (local address 127.0.98.3:45655)
0924 03:02:31.618334 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:31.618376 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:31.618900 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:31.619182 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:31.619185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:31.619878 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:31.620112 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:31.620118 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:31.620808 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:31.620809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:31.621638 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:31.621640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:31.622058 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:31.622059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:31.622180 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:31.622719 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:31.622729 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:31.624147 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:31.625821 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:31.625827 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:31.625829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:31.626136 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:31.626330 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:31.626331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:31.626332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:31.626442 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:31.626710 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:31.626716 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:31.626862 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 03:02:31.626971 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 03:02:31.627142 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
I20260924 03:02:31.627873 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44417
I20260924 03:02:31.627951 903 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:31.628291 903 heartbeater.cc:511] Master 127.0.98.62:44417 requested a full tablet report, sending...
I20260924 03:02:31.628690 444 ts_manager.cc:194] Registered new tserver with Master: 9e8f50538c124032920dc01aaa34853d (127.0.98.3:39243)
I20260924 03:02:31.629062 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45655
I20260924 03:02:31.637744 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 03:02:31 dist-test-slave-wgsb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218951, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 03:02:31 dist-test-slave-wgsb krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218951, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 03:02:31.660723 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:31.652900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41984 (local address 127.0.98.1:46031)
0924 03:02:31.653121 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:31.653127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:31.653139 (+ 12us) server_negotiation.cc:407] Connection header received
0924 03:02:31.653187 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:31.653189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:31.653202 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:31.653270 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:31.653846 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:31.654636 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:31.655371 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:31.655839 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:31.655901 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:31.658171 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:31.658177 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:31.658177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:31.658184 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:31.659667 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:31.659959 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:31.659962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:31.659962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:31.660020 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:31.660176 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:31.660177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:31.660177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:31.660297 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:31.660405 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:31.660460 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 03:02:31.660598 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":130,"threads_started":1}
W20260924 03:02:31.661085 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41984
Sep 24 03:02:31 dist-test-slave-wgsb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 03:02:31 dist-test-slave-wgsb krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 03:02:31.683586 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:31.676154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41990 (local address 127.0.98.1:46031)
0924 03:02:31.676248 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:31.676250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:31.676262 (+ 12us) server_negotiation.cc:407] Connection header received
0924 03:02:31.676402 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:31.676403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:31.676417 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:31.676478 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:31.677131 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:31.677633 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:31.678352 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:31.678540 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:31.678591 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:31.680972 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:31.680973 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:31.680973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:31.680981 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:31.682347 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:31.682688 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:31.682689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:31.682690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:31.682755 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:31.682971 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:31.682972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:31.682972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:31.683089 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:31.683193 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:31.683350 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 03:02:31.683428 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:02:31.684360 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 506
I20260924 03:02:31.690286 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 640
I20260924 03:02:31.696857 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 774
I20260924 03:02:31.701432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 412
2026-09-24T03:02:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3421 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:02:31 dist-test-slave-wgsb krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:02:31 dist-test-slave-wgsb krb5kdc[917](info): set up 2 sockets
Sep 24 03:02:31 dist-test-slave-wgsb krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260924 03:02:32.787282 392 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:02:32 dist-test-slave-wgsb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:02:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:02:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:02:32.911120 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39287
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:32.991947 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:32.992208 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:32.992247 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:32.993413 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:02:32.993482 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:32.993505 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:32.993526 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:02:32.993556 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:02:32.995050 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:32.995328 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:32.995563 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:33.000403 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:33.000432 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:33.000521 930 server_base.cc:1034] running on GCE node
W20260924 03:02:33.000407 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:33.000892 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:33.001132 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:33.002287 930 hybrid_clock.cc:648] HybridClock initialized: now 1790218953002249 us; error 41 us; skew 500 ppm
Sep 24 03:02:33 dist-test-slave-wgsb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:33.005152 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:02:33.005605 930 webserver.cc:460] Webserver started at http://127.0.98.62:43539/ using document root <none> and password file <none>
I20260924 03:02:33.005834 930 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:33.005878 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:33.005967 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:33.006945 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "aeec5d5c3bfe46ebaea0966912c5884e"
format_stamp: "Formatted at 2026-09-24 03:02:33 on dist-test-slave-wgsb"
server_key: "13f59d359298396e535afe82e044e12e"
server_key_iv: "e384176282563523168aab74decacd4c"
server_key_version: "encryptionkey@0"
I20260924 03:02:33.007232 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "aeec5d5c3bfe46ebaea0966912c5884e"
format_stamp: "Formatted at 2026-09-24 03:02:33 on dist-test-slave-wgsb"
server_key: "13f59d359298396e535afe82e044e12e"
server_key_iv: "e384176282563523168aab74decacd4c"
server_key_version: "encryptionkey@0"
I20260924 03:02:33.008569 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:02:33.009270 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:33.009567 930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:02:33.009670 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "aeec5d5c3bfe46ebaea0966912c5884e"
format_stamp: "Formatted at 2026-09-24 03:02:33 on dist-test-slave-wgsb"
server_key: "13f59d359298396e535afe82e044e12e"
server_key_iv: "e384176282563523168aab74decacd4c"
server_key_version: "encryptionkey@0"
I20260924 03:02:33.009725 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:33.019121 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:33.021183 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:33.021291 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:33.025419 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39287
I20260924 03:02:33.025484 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39287 every 8 connection(s)
I20260924 03:02:33.025821 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:02:33.026737 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:33.029000 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeec5d5c3bfe46ebaea0966912c5884e: Bootstrap starting.
I20260924 03:02:33.029536 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aeec5d5c3bfe46ebaea0966912c5884e: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:33.029775 998 log.cc:824] T 00000000000000000000000000000000 P aeec5d5c3bfe46ebaea0966912c5884e: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:33.030423 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeec5d5c3bfe46ebaea0966912c5884e: No bootstrap required, opened a new log
I20260924 03:02:33.032269 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P aeec5d5c3bfe46ebaea0966912c5884e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeec5d5c3bfe46ebaea0966912c5884e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39287 } }
I20260924 03:02:33.032437 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P aeec5d5c3bfe46ebaea0966912c5884e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:33.032490 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P aeec5d5c3bfe46ebaea0966912c5884e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeec5d5c3bfe46ebaea0966912c5884e, State: Initialized, Role: FOLLOWER
I20260924 03:02:33.032651 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P aeec5d5c3bfe46ebaea0966912c5884e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeec5d5c3bfe46ebaea0966912c5884e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39287 } }
I20260924 03:02:33.032739 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P aeec5d5c3bfe46ebaea0966912c5884e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:02:33.032783 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P aeec5d5c3bfe46ebaea0966912c5884e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:02:33.032833 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aeec5d5c3bfe46ebaea0966912c5884e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:33.033447 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P aeec5d5c3bfe46ebaea0966912c5884e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeec5d5c3bfe46ebaea0966912c5884e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39287 } }
I20260924 03:02:33.033571 998 leader_election.cc:302] T 00000000000000000000000000000000 P aeec5d5c3bfe46ebaea0966912c5884e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aeec5d5c3bfe46ebaea0966912c5884e; no voters:
I20260924 03:02:33.033767 998 leader_election.cc:288] T 00000000000000000000000000000000 P aeec5d5c3bfe46ebaea0966912c5884e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:02:33.033815 1001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aeec5d5c3bfe46ebaea0966912c5884e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:33.034036 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P aeec5d5c3bfe46ebaea0966912c5884e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:02:33.034191 1001 raft_consensus.cc:686] T 00000000000000000000000000000000 P aeec5d5c3bfe46ebaea0966912c5884e [term 1 LEADER]: Becoming Leader. State: Replica: aeec5d5c3bfe46ebaea0966912c5884e, State: Running, Role: LEADER
I20260924 03:02:33.034296 1001 consensus_queue.cc:237] T 00000000000000000000000000000000 P aeec5d5c3bfe46ebaea0966912c5884e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeec5d5c3bfe46ebaea0966912c5884e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39287 } }
I20260924 03:02:33.034703 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeec5d5c3bfe46ebaea0966912c5884e [sys.catalog]: SysCatalogTable state changed. Reason: New leader aeec5d5c3bfe46ebaea0966912c5884e. Latest consensus state: current_term: 1 leader_uuid: "aeec5d5c3bfe46ebaea0966912c5884e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeec5d5c3bfe46ebaea0966912c5884e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39287 } } }
I20260924 03:02:33.034804 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeec5d5c3bfe46ebaea0966912c5884e [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:33.034714 1002 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeec5d5c3bfe46ebaea0966912c5884e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aeec5d5c3bfe46ebaea0966912c5884e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeec5d5c3bfe46ebaea0966912c5884e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39287 } } }
I20260924 03:02:33.034840 1002 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeec5d5c3bfe46ebaea0966912c5884e [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:33.035050 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:02:33.035626 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:02:33.035794 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 930
I20260924 03:02:33.035880 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:02:33.036038 392 external_mini_cluster.cc:1428] Setting key 39dfb71fb8b213447970d4a8ca6ecb04
I20260924 03:02:33.037395 1010 catalog_manager.cc:1347] Generated new cluster ID: cf15cc77d8cf4a859f0a3a3949303563
I20260924 03:02:33.037447 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:02:33 dist-test-slave-wgsb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:33.050704 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:02:33.051321 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:02:33.051618 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:33.036628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41890 (local address 127.0.98.62:39287)
0924 03:02:33.036794 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:33.036804 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:33.039835 (+ 3031us) server_negotiation.cc:407] Connection header received
0924 03:02:33.039963 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:33.039971 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:33.040026 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:33.040172 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:33.040935 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.042857 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.043951 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.044596 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.044696 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:33.047581 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:33.047587 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:33.047588 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:33.047638 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:33.049563 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:33.049956 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:33.049959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:33.049960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:33.050071 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:33.050482 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:33.050482 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:33.050483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:33.050776 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:33.050924 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:33.051166 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 03:02:33.051320 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
I20260924 03:02:33.054531 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aeec5d5c3bfe46ebaea0966912c5884e: Generated new TSK 0
I20260924 03:02:33.054720 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:02:33.086876 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39287
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:33.178802 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:33.179044 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:33.179086 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:33.180388 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:33.180457 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:33.180501 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:02:33.182121 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39287
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:33.182406 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:33.182642 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:33.187625 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:33.187652 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:33.187624 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:33.187879 1024 server_base.cc:1034] running on GCE node
I20260924 03:02:33.188067 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:33.188310 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:33.189455 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790218953189445 us; error 26 us; skew 500 ppm
Sep 24 03:02:33 dist-test-slave-wgsb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:33.192027 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:02:33.192543 1024 webserver.cc:460] Webserver started at http://127.0.98.1:36601/ using document root <none> and password file <none>
I20260924 03:02:33.192780 1024 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:33.192849 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:33.193007 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:33.194078 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-0/data/instance:
uuid: "a3d61fe43f144f1db2aad1cc03c83792"
format_stamp: "Formatted at 2026-09-24 03:02:33 on dist-test-slave-wgsb"
server_key: "4c46dba2db5e27e134d52dd7f5afe0d3"
server_key_iv: "2fd1dc15092b551b6918976cd1c7800b"
server_key_version: "encryptionkey@0"
I20260924 03:02:33.194449 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a3d61fe43f144f1db2aad1cc03c83792"
format_stamp: "Formatted at 2026-09-24 03:02:33 on dist-test-slave-wgsb"
server_key: "4c46dba2db5e27e134d52dd7f5afe0d3"
server_key_iv: "2fd1dc15092b551b6918976cd1c7800b"
server_key_version: "encryptionkey@0"
I20260924 03:02:33.195850 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:02:33.196557 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:33.196817 1024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:33.196921 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "a3d61fe43f144f1db2aad1cc03c83792"
format_stamp: "Formatted at 2026-09-24 03:02:33 on dist-test-slave-wgsb"
server_key: "4c46dba2db5e27e134d52dd7f5afe0d3"
server_key_iv: "2fd1dc15092b551b6918976cd1c7800b"
server_key_version: "encryptionkey@0"
I20260924 03:02:33.196995 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:33.218367 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:33.220561 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:33.220753 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:33.221019 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:33.221400 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:33.221459 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:33.221506 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:33.221539 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:33.228673 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33191
I20260924 03:02:33.228708 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33191 every 8 connection(s)
I20260924 03:02:33.229029 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:02:33.232553 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 1024
I20260924 03:02:33.232645 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:02:33.232808 392 external_mini_cluster.cc:1428] Setting key 666cf188f1740dcb1eff07fddf85caf9
Sep 24 03:02:33 dist-test-slave-wgsb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:33.239343 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:33.229749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57489 (local address 127.0.98.62:39287)
0924 03:02:33.229830 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:33.229832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:33.230012 (+ 180us) server_negotiation.cc:407] Connection header received
0924 03:02:33.230588 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:33.230589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:33.230602 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:33.230652 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:33.231652 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.232178 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.233652 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.233861 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.233916 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:33.236481 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:33.236482 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:33.236483 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:33.236490 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:33.237938 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:33.238371 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:33.238372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:33.238373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:33.238429 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:33.238683 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:33.238684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:33.238684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:33.238811 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:33.238888 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:33.239159 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 03:02:33.239250 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 03:02:33.239558 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:33.229657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39287 (local address 127.0.98.1:57489)
0924 03:02:33.229875 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:33.229928 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:33.230464 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:33.230725 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:33.230729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:33.231319 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:33.231511 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:33.231516 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.232286 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.232287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:33.233562 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:33.233565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.233965 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.233966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:33.234132 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:33.234746 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:33.234756 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:33.236382 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:33.238042 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:33.238048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:33.238050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:33.238281 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:33.238503 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:33.238504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:33.238504 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:33.238615 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:33.238879 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:33.238885 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:33.239029 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 03:02:33.239139 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 03:02:33.239307 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":103,"threads_started":1}
I20260924 03:02:33.239972 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39287
I20260924 03:02:33.240073 1153 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:33.240280 1153 heartbeater.cc:511] Master 127.0.98.62:39287 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 03:02:33.240813 962 ts_manager.cc:194] Registered new tserver with Master: a3d61fe43f144f1db2aad1cc03c83792 (127.0.98.1:33191)
I20260924 03:02:33.241433 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:02:33.260288 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39287
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:33.341897 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:33.342118 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:33.342144 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:33.343286 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:33.343328 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:33.343357 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:02:33.344934 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39287
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:33.345162 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:33.345355 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:33.351708 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:33.351727 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:33.351926 1158 server_base.cc:1034] running on GCE node
W20260924 03:02:33.351809 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:33.352200 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:33.352481 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:33.353647 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790218953353628 us; error 33 us; skew 500 ppm
Sep 24 03:02:33 dist-test-slave-wgsb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:33.356884 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:02:33.357415 1158 webserver.cc:460] Webserver started at http://127.0.98.2:42295/ using document root <none> and password file <none>
I20260924 03:02:33.357690 1158 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:33.357774 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:33.357941 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:33.359166 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-1/data/instance:
uuid: "effa757782a546c89d096b6df1c0019d"
format_stamp: "Formatted at 2026-09-24 03:02:33 on dist-test-slave-wgsb"
server_key: "d257c9be982ea71e5963ef4d9432e55a"
server_key_iv: "1a55ab79ed3d51dc2fd9a2fd2bd38876"
server_key_version: "encryptionkey@0"
I20260924 03:02:33.359578 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-1/wal/instance:
uuid: "effa757782a546c89d096b6df1c0019d"
format_stamp: "Formatted at 2026-09-24 03:02:33 on dist-test-slave-wgsb"
server_key: "d257c9be982ea71e5963ef4d9432e55a"
server_key_iv: "1a55ab79ed3d51dc2fd9a2fd2bd38876"
server_key_version: "encryptionkey@0"
I20260924 03:02:33.361130 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:02:33.361974 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:33.362146 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:02:33.362246 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "effa757782a546c89d096b6df1c0019d"
format_stamp: "Formatted at 2026-09-24 03:02:33 on dist-test-slave-wgsb"
server_key: "d257c9be982ea71e5963ef4d9432e55a"
server_key_iv: "1a55ab79ed3d51dc2fd9a2fd2bd38876"
server_key_version: "encryptionkey@0"
I20260924 03:02:33.362332 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:33.376080 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:33.378461 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:33.378660 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:33.378923 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:33.379264 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:33.379325 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:33.379359 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:33.379390 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:33.385553 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42915
I20260924 03:02:33.385618 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42915 every 8 connection(s)
I20260924 03:02:33.385983 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 03:02:33 dist-test-slave-wgsb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:33.395219 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 1158
I20260924 03:02:33.395339 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:02:33.395520 392 external_mini_cluster.cc:1428] Setting key f87de394b2048d347349c567be18cf70
I20260924 03:02:33.396028 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:33.386774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34613 (local address 127.0.98.62:39287)
0924 03:02:33.386868 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:33.386871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:33.387053 (+ 182us) server_negotiation.cc:407] Connection header received
0924 03:02:33.387576 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:33.387577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:33.387622 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:33.387745 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:33.388752 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.389272 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.390303 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.390516 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.390604 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:33.392965 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:33.392966 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:33.392967 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:33.392974 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:33.394499 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:33.394956 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:33.394957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:33.394957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:33.395017 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:33.395277 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:33.395278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:33.395279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:33.395443 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:33.395506 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:33.395850 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 03:02:33.395936 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:02:33.396308 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:33.386742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39287 (local address 127.0.98.2:34613)
0924 03:02:33.386922 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:33.386960 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:33.387461 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:33.387831 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:33.387835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:33.388444 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:33.388647 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:33.388652 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.389371 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.389372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:33.390197 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:33.390199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.390611 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.390612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:33.390738 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:33.391228 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:33.391236 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:33.392864 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:33.394613 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:33.394620 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:33.394622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:33.394863 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:33.395084 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:33.395085 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:33.395086 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:33.395207 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:33.395496 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:33.395502 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:33.395715 (+ 213us) client_negotiation.cc:769] Sending connection context
0924 03:02:33.395829 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 03:02:33.395990 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":91,"threads_started":1}
I20260924 03:02:33.396909 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39287
I20260924 03:02:33.397020 1287 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:33.397215 1287 heartbeater.cc:511] Master 127.0.98.62:39287 requested a full tablet report, sending...
I20260924 03:02:33.397600 962 ts_manager.cc:194] Registered new tserver with Master: effa757782a546c89d096b6df1c0019d (127.0.98.2:42915)
I20260924 03:02:33.398015 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34613
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:02:33.423250 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39287
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:33.503826 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:33.504074 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:33.504112 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:33.505286 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:33.505354 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:33.505396 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:02:33.506816 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39287
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:33.507086 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:33.507326 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:33.512355 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:33.512427 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:33.512455 1292 server_base.cc:1034] running on GCE node
W20260924 03:02:33.512501 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:33.512813 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:33.513067 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:33.514217 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790218953514182 us; error 51 us; skew 500 ppm
Sep 24 03:02:33 dist-test-slave-wgsb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:33.516755 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:02:33.517181 1292 webserver.cc:460] Webserver started at http://127.0.98.3:35515/ using document root <none> and password file <none>
I20260924 03:02:33.517429 1292 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:33.517499 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:33.517654 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:33.518692 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-2/data/instance:
uuid: "ff3dd58fe3644d698784ab655332d28f"
format_stamp: "Formatted at 2026-09-24 03:02:33 on dist-test-slave-wgsb"
server_key: "d33ef258c832fae53ab3071fd7c8968b"
server_key_iv: "c02e9bb231539e8e6bee4f8a9f5f610a"
server_key_version: "encryptionkey@0"
I20260924 03:02:33.519055 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ff3dd58fe3644d698784ab655332d28f"
format_stamp: "Formatted at 2026-09-24 03:02:33 on dist-test-slave-wgsb"
server_key: "d33ef258c832fae53ab3071fd7c8968b"
server_key_iv: "c02e9bb231539e8e6bee4f8a9f5f610a"
server_key_version: "encryptionkey@0"
I20260924 03:02:33.520416 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:02:33.521117 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:33.521291 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:02:33.521389 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "ff3dd58fe3644d698784ab655332d28f"
format_stamp: "Formatted at 2026-09-24 03:02:33 on dist-test-slave-wgsb"
server_key: "d33ef258c832fae53ab3071fd7c8968b"
server_key_iv: "c02e9bb231539e8e6bee4f8a9f5f610a"
server_key_version: "encryptionkey@0"
I20260924 03:02:33.521471 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:33.547786 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:33.550151 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:33.550261 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:33.550452 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:33.550691 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:33.550719 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:33.550741 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:33.550761 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:33.556903 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42669
I20260924 03:02:33.556945 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42669 every 8 connection(s)
I20260924 03:02:33.557253 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:02:33.558233 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 1292
I20260924 03:02:33.558336 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218948286999-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:02:33.558492 392 external_mini_cluster.cc:1428] Setting key f914d872e218d0cf10992d35fde2bca1
Sep 24 03:02:33 dist-test-slave-wgsb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:33.566809 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:33.557944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44975 (local address 127.0.98.62:39287)
0924 03:02:33.558007 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:33.558009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:33.558224 (+ 215us) server_negotiation.cc:407] Connection header received
0924 03:02:33.558871 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:33.558871 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:33.558883 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:33.558921 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:33.560017 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.560516 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.561514 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.561720 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.561802 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:33.563907 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:33.563908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:33.563909 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:33.563916 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:33.565366 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:33.565846 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:33.565847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:33.565848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:33.565909 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:33.566158 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:33.566159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:33.566159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:33.566291 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:33.566369 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:33.566636 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 03:02:33.566714 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 03:02:33.567013 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:33.557937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39287 (local address 127.0.98.3:44975)
0924 03:02:33.558106 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:33.558147 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:33.558786 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:33.559001 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:33.559005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:33.559715 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:33.559916 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:33.559921 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.560606 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.560607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:33.561412 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:33.561414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.561806 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.561807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:33.561941 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:33.562377 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:33.562385 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:33.563775 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:33.565476 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:33.565482 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:33.565484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:33.565736 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:33.565975 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:33.565976 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:33.565977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:33.566089 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:33.566360 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:33.566366 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:33.566504 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 03:02:33.566613 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 03:02:33.566770 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260924 03:02:33.567440 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39287
I20260924 03:02:33.567560 1421 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:33.567786 1421 heartbeater.cc:511] Master 127.0.98.62:39287 requested a full tablet report, sending...
I20260924 03:02:33.568153 962 ts_manager.cc:194] Registered new tserver with Master: ff3dd58fe3644d698784ab655332d28f (127.0.98.3:42669)
I20260924 03:02:33.568511 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44975
I20260924 03:02:33.570225 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:02:33.577247 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:33.571707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41892 (local address 127.0.98.62:39287)
0924 03:02:33.571775 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:33.571778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:33.571906 (+ 128us) server_negotiation.cc:407] Connection header received
0924 03:02:33.571944 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:33.571945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:33.571953 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:33.571999 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:33.572507 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.572968 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.573660 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.573831 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.573877 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:33.574990 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:33.574990 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:33.574991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:33.574995 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:33.576328 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:33.576632 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:33.576633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:33.576633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:33.576679 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:33.576835 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:33.576835 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:33.576835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:33.576955 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:33.577026 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:33.577108 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 03:02:33.577159 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 03:02:33.579293 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:02:33.580941 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:02:33.587634 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:33.583325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33191 (local address 127.0.0.1:58292)
0924 03:02:33.583647 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:33.583689 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:33.583759 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:33.584066 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:33.584069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:33.584071 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:33.584136 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:33.584138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.585528 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.585529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:33.586598 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:33.586600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.587268 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.587269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:33.587354 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:33.587367 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 03:02:33.587471 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 03:02:33.587526 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":25,"thread_start_us":48,"threads_started":1}
I20260924 03:02:33.587627 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:33.583451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58292 (local address 127.0.98.1:33191)
0924 03:02:33.583919 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:33.583923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:33.583932 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:02:33.583963 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:33.583965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:33.583977 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:33.584064 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:33.584222 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.585428 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.586681 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.587166 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.587265 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:33.587326 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:33.587489 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 03:02:33.587531 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":77,"threads_started":1}
I20260924 03:02:33.587824 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:33.583436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48814 (local address 127.0.98.3:42669)
0924 03:02:33.583948 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:33.583952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:33.583961 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:02:33.583994 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:33.583996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:33.584011 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:33.584082 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:33.584443 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.585504 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.586584 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.587282 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.587353 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:33.587405 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:33.587680 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 03:02:33.587723 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":76,"threads_started":1}
I20260924 03:02:33.588016 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:33.583642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52022 (local address 127.0.98.2:42915)
0924 03:02:33.583823 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:33.583828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:33.583855 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:02:33.583899 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:33.583901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:33.583916 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:33.584326 (+ 410us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:33.584407 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.585082 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.586537 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.587282 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.587353 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:33.587406 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:33.587831 (+ 425us) server_negotiation.cc:299] Negotiation successful
0924 03:02:33.587942 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":78,"threads_started":1}
I20260924 03:02:33.588023 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:33.583507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42915 (local address 127.0.0.1:52022)
0924 03:02:33.583751 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:33.583763 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:33.583821 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:33.584162 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:33.584162 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:33.584164 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:33.584205 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:33.584206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.585220 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.585221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:33.586444 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:33.586447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.587442 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.587444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:33.587515 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:33.587519 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:02:33.587568 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 03:02:33.587637 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":66,"threads_started":1}
I20260924 03:02:33.588224 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:33.583325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42669 (local address 127.0.0.1:48814)
0924 03:02:33.583647 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:33.583689 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:33.583759 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:33.584166 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:33.584167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:33.584169 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:33.584341 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:33.584344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.585588 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.585589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:33.586475 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:33.586477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.587444 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.587445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:33.587515 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:33.587520 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:02:33.587568 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 03:02:33.587637 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":10,"thread_start_us":98,"threads_started":1}
I20260924 03:02:33.588431 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 0891fc76b3ae44f884a5fb4fe36bbe5c (DEFAULT_TABLE table=test-table [id=4570f3432a064d1f90485a9e52fbf8f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:33.588830 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0891fc76b3ae44f884a5fb4fe36bbe5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:33.588935 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 0891fc76b3ae44f884a5fb4fe36bbe5c (DEFAULT_TABLE table=test-table [id=4570f3432a064d1f90485a9e52fbf8f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:33.589251 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0891fc76b3ae44f884a5fb4fe36bbe5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:33.591575 1438 tablet_bootstrap.cc:492] T 0891fc76b3ae44f884a5fb4fe36bbe5c P a3d61fe43f144f1db2aad1cc03c83792: Bootstrap starting.
I20260924 03:02:33.592025 1439 tablet_bootstrap.cc:492] T 0891fc76b3ae44f884a5fb4fe36bbe5c P effa757782a546c89d096b6df1c0019d: Bootstrap starting.
I20260924 03:02:33.592118 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 0891fc76b3ae44f884a5fb4fe36bbe5c (DEFAULT_TABLE table=test-table [id=4570f3432a064d1f90485a9e52fbf8f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:33.592293 1438 tablet_bootstrap.cc:654] T 0891fc76b3ae44f884a5fb4fe36bbe5c P a3d61fe43f144f1db2aad1cc03c83792: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:33.592453 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0891fc76b3ae44f884a5fb4fe36bbe5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:33.592679 1439 tablet_bootstrap.cc:654] T 0891fc76b3ae44f884a5fb4fe36bbe5c P effa757782a546c89d096b6df1c0019d: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:33.592942 1438 log.cc:824] T 0891fc76b3ae44f884a5fb4fe36bbe5c P a3d61fe43f144f1db2aad1cc03c83792: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:33.593237 1439 log.cc:824] T 0891fc76b3ae44f884a5fb4fe36bbe5c P effa757782a546c89d096b6df1c0019d: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:33.593894 1439 tablet_bootstrap.cc:492] T 0891fc76b3ae44f884a5fb4fe36bbe5c P effa757782a546c89d096b6df1c0019d: No bootstrap required, opened a new log
I20260924 03:02:33.593976 1439 ts_tablet_manager.cc:1397] T 0891fc76b3ae44f884a5fb4fe36bbe5c P effa757782a546c89d096b6df1c0019d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:02:33.594137 1438 tablet_bootstrap.cc:492] T 0891fc76b3ae44f884a5fb4fe36bbe5c P a3d61fe43f144f1db2aad1cc03c83792: No bootstrap required, opened a new log
I20260924 03:02:33.594211 1438 ts_tablet_manager.cc:1397] T 0891fc76b3ae44f884a5fb4fe36bbe5c P a3d61fe43f144f1db2aad1cc03c83792: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 03:02:33.595345 1439 raft_consensus.cc:348] T 0891fc76b3ae44f884a5fb4fe36bbe5c P effa757782a546c89d096b6df1c0019d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d61fe43f144f1db2aad1cc03c83792" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33191 } } peers { permanent_uuid: "ff3dd58fe3644d698784ab655332d28f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42669 } } peers { permanent_uuid: "effa757782a546c89d096b6df1c0019d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } }
I20260924 03:02:33.595507 1439 raft_consensus.cc:374] T 0891fc76b3ae44f884a5fb4fe36bbe5c P effa757782a546c89d096b6df1c0019d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:33.595551 1439 raft_consensus.cc:729] T 0891fc76b3ae44f884a5fb4fe36bbe5c P effa757782a546c89d096b6df1c0019d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: effa757782a546c89d096b6df1c0019d, State: Initialized, Role: FOLLOWER
I20260924 03:02:33.595722 1439 consensus_queue.cc:260] T 0891fc76b3ae44f884a5fb4fe36bbe5c P effa757782a546c89d096b6df1c0019d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d61fe43f144f1db2aad1cc03c83792" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33191 } } peers { permanent_uuid: "ff3dd58fe3644d698784ab655332d28f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42669 } } peers { permanent_uuid: "effa757782a546c89d096b6df1c0019d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } }
I20260924 03:02:33.596006 1439 ts_tablet_manager.cc:1428] T 0891fc76b3ae44f884a5fb4fe36bbe5c P effa757782a546c89d096b6df1c0019d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:02:33.596006 1287 heartbeater.cc:503] Master 127.0.98.62:39287 was elected leader, sending a full tablet report...
I20260924 03:02:33.596205 1438 raft_consensus.cc:348] T 0891fc76b3ae44f884a5fb4fe36bbe5c P a3d61fe43f144f1db2aad1cc03c83792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d61fe43f144f1db2aad1cc03c83792" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33191 } } peers { permanent_uuid: "ff3dd58fe3644d698784ab655332d28f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42669 } } peers { permanent_uuid: "effa757782a546c89d096b6df1c0019d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } }
I20260924 03:02:33.596395 1438 raft_consensus.cc:374] T 0891fc76b3ae44f884a5fb4fe36bbe5c P a3d61fe43f144f1db2aad1cc03c83792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:33.596444 1438 raft_consensus.cc:729] T 0891fc76b3ae44f884a5fb4fe36bbe5c P a3d61fe43f144f1db2aad1cc03c83792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3d61fe43f144f1db2aad1cc03c83792, State: Initialized, Role: FOLLOWER
I20260924 03:02:33.596557 1438 consensus_queue.cc:260] T 0891fc76b3ae44f884a5fb4fe36bbe5c P a3d61fe43f144f1db2aad1cc03c83792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d61fe43f144f1db2aad1cc03c83792" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33191 } } peers { permanent_uuid: "ff3dd58fe3644d698784ab655332d28f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42669 } } peers { permanent_uuid: "effa757782a546c89d096b6df1c0019d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } }
I20260924 03:02:33.596685 1443 tablet_bootstrap.cc:492] T 0891fc76b3ae44f884a5fb4fe36bbe5c P ff3dd58fe3644d698784ab655332d28f: Bootstrap starting.
I20260924 03:02:33.596812 1438 ts_tablet_manager.cc:1428] T 0891fc76b3ae44f884a5fb4fe36bbe5c P a3d61fe43f144f1db2aad1cc03c83792: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 03:02:33.596860 1153 heartbeater.cc:503] Master 127.0.98.62:39287 was elected leader, sending a full tablet report...
I20260924 03:02:33.597402 1443 tablet_bootstrap.cc:654] T 0891fc76b3ae44f884a5fb4fe36bbe5c P ff3dd58fe3644d698784ab655332d28f: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:33.597884 1443 log.cc:824] T 0891fc76b3ae44f884a5fb4fe36bbe5c P ff3dd58fe3644d698784ab655332d28f: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:33.598546 1443 tablet_bootstrap.cc:492] T 0891fc76b3ae44f884a5fb4fe36bbe5c P ff3dd58fe3644d698784ab655332d28f: No bootstrap required, opened a new log
I20260924 03:02:33.598640 1443 ts_tablet_manager.cc:1397] T 0891fc76b3ae44f884a5fb4fe36bbe5c P ff3dd58fe3644d698784ab655332d28f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:02:33.600107 1443 raft_consensus.cc:348] T 0891fc76b3ae44f884a5fb4fe36bbe5c P ff3dd58fe3644d698784ab655332d28f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d61fe43f144f1db2aad1cc03c83792" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33191 } } peers { permanent_uuid: "ff3dd58fe3644d698784ab655332d28f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42669 } } peers { permanent_uuid: "effa757782a546c89d096b6df1c0019d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } }
I20260924 03:02:33.600283 1443 raft_consensus.cc:374] T 0891fc76b3ae44f884a5fb4fe36bbe5c P ff3dd58fe3644d698784ab655332d28f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:33.600327 1443 raft_consensus.cc:729] T 0891fc76b3ae44f884a5fb4fe36bbe5c P ff3dd58fe3644d698784ab655332d28f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff3dd58fe3644d698784ab655332d28f, State: Initialized, Role: FOLLOWER
I20260924 03:02:33.600440 1443 consensus_queue.cc:260] T 0891fc76b3ae44f884a5fb4fe36bbe5c P ff3dd58fe3644d698784ab655332d28f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d61fe43f144f1db2aad1cc03c83792" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33191 } } peers { permanent_uuid: "ff3dd58fe3644d698784ab655332d28f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42669 } } peers { permanent_uuid: "effa757782a546c89d096b6df1c0019d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } }
I20260924 03:02:33.600710 1443 ts_tablet_manager.cc:1428] T 0891fc76b3ae44f884a5fb4fe36bbe5c P ff3dd58fe3644d698784ab655332d28f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:02:33.600723 1421 heartbeater.cc:503] Master 127.0.98.62:39287 was elected leader, sending a full tablet report...
W20260924 03:02:33.636562 1288 tablet.cc:2369] T 0891fc76b3ae44f884a5fb4fe36bbe5c P effa757782a546c89d096b6df1c0019d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:02:33.709436 1444 raft_consensus.cc:482] T 0891fc76b3ae44f884a5fb4fe36bbe5c P a3d61fe43f144f1db2aad1cc03c83792 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:02:33.709550 1444 raft_consensus.cc:504] T 0891fc76b3ae44f884a5fb4fe36bbe5c P a3d61fe43f144f1db2aad1cc03c83792 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d61fe43f144f1db2aad1cc03c83792" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33191 } } peers { permanent_uuid: "ff3dd58fe3644d698784ab655332d28f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42669 } } peers { permanent_uuid: "effa757782a546c89d096b6df1c0019d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } }
I20260924 03:02:33.709978 1444 leader_election.cc:288] T 0891fc76b3ae44f884a5fb4fe36bbe5c P a3d61fe43f144f1db2aad1cc03c83792 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff3dd58fe3644d698784ab655332d28f (127.0.98.3:42669), effa757782a546c89d096b6df1c0019d (127.0.98.2:42915)
I20260924 03:02:33.712999 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:33.710140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42915 (local address 127.0.98.1:49947)
0924 03:02:33.710277 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:33.710289 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:33.710353 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:33.710492 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:33.710492 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:33.710494 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:33.710569 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:33.710570 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.711199 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.711200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:33.712112 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:33.712115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.712713 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.712714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:33.712817 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:33.712821 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:02:33.712879 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 03:02:33.712918 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260924 03:02:33.712991 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:33.710238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49947 (local address 127.0.98.2:42915)
0924 03:02:33.710302 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:33.710305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:33.710342 (+ 37us) server_negotiation.cc:407] Connection header received
0924 03:02:33.710427 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:33.710428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:33.710439 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:33.710489 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:33.710640 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.711108 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.712194 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.712628 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.712703 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:33.712714 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:33.712888 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 03:02:33.712928 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 03:02:33.713189 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:33.710123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42669 (local address 127.0.98.1:54117)
0924 03:02:33.710182 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:33.710194 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:33.710237 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:33.710392 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:33.710392 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:33.710394 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:33.710466 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:33.710468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.711121 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.711122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:33.711883 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:33.711885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.712902 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.712903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:33.712978 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:33.712980 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:02:33.713055 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 03:02:33.713107 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 03:02:33.713299 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:33.710202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54117 (local address 127.0.98.3:42669)
0924 03:02:33.710254 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:33.710256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:33.710266 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:02:33.710292 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:33.710294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:33.710304 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:33.710385 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:33.710537 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.711032 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.711986 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.712821 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.712903 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:33.712912 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:33.713179 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 03:02:33.713223 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 03:02:33.713515 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0891fc76b3ae44f884a5fb4fe36bbe5c" candidate_uuid: "a3d61fe43f144f1db2aad1cc03c83792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "effa757782a546c89d096b6df1c0019d" is_pre_election: true
I20260924 03:02:33.713574 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0891fc76b3ae44f884a5fb4fe36bbe5c" candidate_uuid: "a3d61fe43f144f1db2aad1cc03c83792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff3dd58fe3644d698784ab655332d28f" is_pre_election: true
I20260924 03:02:33.713680 1241 raft_consensus.cc:2454] T 0891fc76b3ae44f884a5fb4fe36bbe5c P effa757782a546c89d096b6df1c0019d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3d61fe43f144f1db2aad1cc03c83792 in term 0.
I20260924 03:02:33.713762 1375 raft_consensus.cc:2454] T 0891fc76b3ae44f884a5fb4fe36bbe5c P ff3dd58fe3644d698784ab655332d28f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3d61fe43f144f1db2aad1cc03c83792 in term 0.
I20260924 03:02:33.713891 1041 leader_election.cc:302] T 0891fc76b3ae44f884a5fb4fe36bbe5c P a3d61fe43f144f1db2aad1cc03c83792 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3d61fe43f144f1db2aad1cc03c83792, effa757782a546c89d096b6df1c0019d; no voters:
I20260924 03:02:33.714027 1444 raft_consensus.cc:2781] T 0891fc76b3ae44f884a5fb4fe36bbe5c P a3d61fe43f144f1db2aad1cc03c83792 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:02:33.714087 1444 raft_consensus.cc:482] T 0891fc76b3ae44f884a5fb4fe36bbe5c P a3d61fe43f144f1db2aad1cc03c83792 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:02:33.714104 1444 raft_consensus.cc:3037] T 0891fc76b3ae44f884a5fb4fe36bbe5c P a3d61fe43f144f1db2aad1cc03c83792 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:33.714795 1444 raft_consensus.cc:504] T 0891fc76b3ae44f884a5fb4fe36bbe5c P a3d61fe43f144f1db2aad1cc03c83792 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d61fe43f144f1db2aad1cc03c83792" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33191 } } peers { permanent_uuid: "ff3dd58fe3644d698784ab655332d28f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42669 } } peers { permanent_uuid: "effa757782a546c89d096b6df1c0019d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } }
I20260924 03:02:33.714963 1444 leader_election.cc:288] T 0891fc76b3ae44f884a5fb4fe36bbe5c P a3d61fe43f144f1db2aad1cc03c83792 [CANDIDATE]: Term 1 election: Requested vote from peers ff3dd58fe3644d698784ab655332d28f (127.0.98.3:42669), effa757782a546c89d096b6df1c0019d (127.0.98.2:42915)
I20260924 03:02:33.715094 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0891fc76b3ae44f884a5fb4fe36bbe5c" candidate_uuid: "a3d61fe43f144f1db2aad1cc03c83792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff3dd58fe3644d698784ab655332d28f"
I20260924 03:02:33.715129 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0891fc76b3ae44f884a5fb4fe36bbe5c" candidate_uuid: "a3d61fe43f144f1db2aad1cc03c83792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "effa757782a546c89d096b6df1c0019d"
I20260924 03:02:33.715165 1375 raft_consensus.cc:3037] T 0891fc76b3ae44f884a5fb4fe36bbe5c P ff3dd58fe3644d698784ab655332d28f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:33.715212 1241 raft_consensus.cc:3037] T 0891fc76b3ae44f884a5fb4fe36bbe5c P effa757782a546c89d096b6df1c0019d [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:33.715941 1375 raft_consensus.cc:2454] T 0891fc76b3ae44f884a5fb4fe36bbe5c P ff3dd58fe3644d698784ab655332d28f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3d61fe43f144f1db2aad1cc03c83792 in term 1.
I20260924 03:02:33.715984 1241 raft_consensus.cc:2454] T 0891fc76b3ae44f884a5fb4fe36bbe5c P effa757782a546c89d096b6df1c0019d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3d61fe43f144f1db2aad1cc03c83792 in term 1.
I20260924 03:02:33.716161 1040 leader_election.cc:302] T 0891fc76b3ae44f884a5fb4fe36bbe5c P a3d61fe43f144f1db2aad1cc03c83792 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3d61fe43f144f1db2aad1cc03c83792, ff3dd58fe3644d698784ab655332d28f; no voters:
I20260924 03:02:33.716255 1444 raft_consensus.cc:2781] T 0891fc76b3ae44f884a5fb4fe36bbe5c P a3d61fe43f144f1db2aad1cc03c83792 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:33.716439 1444 raft_consensus.cc:686] T 0891fc76b3ae44f884a5fb4fe36bbe5c P a3d61fe43f144f1db2aad1cc03c83792 [term 1 LEADER]: Becoming Leader. State: Replica: a3d61fe43f144f1db2aad1cc03c83792, State: Running, Role: LEADER
I20260924 03:02:33.716530 1444 consensus_queue.cc:237] T 0891fc76b3ae44f884a5fb4fe36bbe5c P a3d61fe43f144f1db2aad1cc03c83792 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d61fe43f144f1db2aad1cc03c83792" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33191 } } peers { permanent_uuid: "ff3dd58fe3644d698784ab655332d28f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42669 } } peers { permanent_uuid: "effa757782a546c89d096b6df1c0019d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } }
I20260924 03:02:33.717340 962 catalog_manager.cc:5495] T 0891fc76b3ae44f884a5fb4fe36bbe5c P a3d61fe43f144f1db2aad1cc03c83792 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3d61fe43f144f1db2aad1cc03c83792 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a3d61fe43f144f1db2aad1cc03c83792" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d61fe43f144f1db2aad1cc03c83792" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff3dd58fe3644d698784ab655332d28f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "effa757782a546c89d096b6df1c0019d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } health_report { overall_health: UNKNOWN } } }
W20260924 03:02:33.729755 1154 tablet.cc:2369] T 0891fc76b3ae44f884a5fb4fe36bbe5c P a3d61fe43f144f1db2aad1cc03c83792: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 03:02:33 dist-test-slave-wgsb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 03:02:33.741847 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:33.730469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58302 (local address 127.0.98.1:33191)
0924 03:02:33.730546 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:33.730548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:33.730746 (+ 198us) server_negotiation.cc:407] Connection header received
0924 03:02:33.730778 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:33.730779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:33.730789 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:33.730852 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:33.731426 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.731963 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.732677 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.732864 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.732962 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:33.739161 (+ 6199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:33.739173 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:33.739173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:33.739188 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:33.740667 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:33.741040 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:33.741043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:33.741044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:33.741101 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:33.741301 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:33.741302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:33.741302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:33.741445 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:33.741609 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:33.741648 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 03:02:33.741728 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 03:02:33 dist-test-slave-wgsb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218953, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 03:02:33 dist-test-slave-wgsb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218953, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 03:02:33.765455 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:33.758080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58306 (local address 127.0.98.1:33191)
0924 03:02:33.758145 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:33.758147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:33.758198 (+ 51us) server_negotiation.cc:407] Connection header received
0924 03:02:33.758257 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:33.758257 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:33.758266 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:33.758352 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:33.758911 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.759398 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.760199 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.760398 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.760451 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:33.762690 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:33.762691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:33.762691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:33.762697 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:33.764171 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:33.764545 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:33.764546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:33.764547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:33.764608 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:33.764872 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:33.764873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:33.764873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:33.764969 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:33.765048 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:33.765308 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 03:02:33.765388 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 03:02:33 dist-test-slave-wgsb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 03:02:33 dist-test-slave-wgsb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 03:02:33.788123 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:33.781234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58310 (local address 127.0.98.1:33191)
0924 03:02:33.781300 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:33.781302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:33.781316 (+ 14us) server_negotiation.cc:407] Connection header received
0924 03:02:33.781425 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:33.781426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:33.781436 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:33.781506 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:33.782081 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.782550 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.783275 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:33.783464 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:33.783574 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:33.785877 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:33.785879 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:33.785879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:33.785885 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:33.787045 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:33.787381 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:33.787381 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:33.787382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:33.787431 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:33.787701 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:33.787702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:33.787703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:33.787819 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:33.787919 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:33.787997 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 03:02:33.788057 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 03:02:33.788599 1067 tablet_service.cc:2908] Rejecting scan request for tablet 0891fc76b3ae44f884a5fb4fe36bbe5c: Uninitialized: clean time has not yet been initialized
I20260924 03:02:33.789476 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 1024
I20260924 03:02:33.794831 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 1158
I20260924 03:02:33.799095 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 1292
I20260924 03:02:33.803484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 930
2026-09-24T03:02:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2102 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:02:33 dist-test-slave-wgsb krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:02:33 dist-test-slave-wgsb krb5kdc[1462](info): set up 2 sockets
Sep 24 03:02:33 dist-test-slave-wgsb krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260924 03:02:34.896057 392 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260924 03:02:35.066932 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.171s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:02:35 dist-test-slave-wgsb krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:02:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:02:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:02:35.232177 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46523
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:35.313732 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:35.313956 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:35.313977 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:35.315068 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:02:35.315104 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 03:02:35.315119 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:35.315130 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:35.315143 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:02:35.315155 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:02:35.316606 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46523
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:35.316883 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:35.317087 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:35.322119 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:35.322126 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:35.322126 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:35.322438 1475 server_base.cc:1034] running on GCE node
I20260924 03:02:35.322691 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:35.322999 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:35.324186 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790218955324174 us; error 44 us; skew 500 ppm
Sep 24 03:02:35 dist-test-slave-wgsb krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:35.326831 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:02:35.327337 1475 webserver.cc:460] Webserver started at http://127.0.98.62:44919/ using document root <none> and password file <none>
I20260924 03:02:35.327559 1475 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:35.327651 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:35.327812 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:35.328888 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "e22f16deebe84552be594e3177fe97b3"
format_stamp: "Formatted at 2026-09-24 03:02:35 on dist-test-slave-wgsb"
server_key: "2e746e4cc5cd3ad323f0745df57c648b"
server_key_iv: "5f8c77050564f8b54041a7993320c012"
server_key_version: "encryptionkey@0"
I20260924 03:02:35.329258 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "e22f16deebe84552be594e3177fe97b3"
format_stamp: "Formatted at 2026-09-24 03:02:35 on dist-test-slave-wgsb"
server_key: "2e746e4cc5cd3ad323f0745df57c648b"
server_key_iv: "5f8c77050564f8b54041a7993320c012"
server_key_version: "encryptionkey@0"
I20260924 03:02:35.330703 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:02:35.331581 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:35.332037 1475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:02:35.332158 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "e22f16deebe84552be594e3177fe97b3"
format_stamp: "Formatted at 2026-09-24 03:02:35 on dist-test-slave-wgsb"
server_key: "2e746e4cc5cd3ad323f0745df57c648b"
server_key_iv: "5f8c77050564f8b54041a7993320c012"
server_key_version: "encryptionkey@0"
I20260924 03:02:35.332237 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:35.341768 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:35.344144 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:35.344321 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:35.349190 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46523
I20260924 03:02:35.349229 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46523 every 8 connection(s)
I20260924 03:02:35.349582 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:02:35.350188 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:35.352617 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e22f16deebe84552be594e3177fe97b3: Bootstrap starting.
I20260924 03:02:35.353229 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e22f16deebe84552be594e3177fe97b3: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:35.353533 1553 log.cc:824] T 00000000000000000000000000000000 P e22f16deebe84552be594e3177fe97b3: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:35.354233 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e22f16deebe84552be594e3177fe97b3: No bootstrap required, opened a new log
I20260924 03:02:35.356451 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P e22f16deebe84552be594e3177fe97b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22f16deebe84552be594e3177fe97b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46523 } }
I20260924 03:02:35.356633 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P e22f16deebe84552be594e3177fe97b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:35.356721 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P e22f16deebe84552be594e3177fe97b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e22f16deebe84552be594e3177fe97b3, State: Initialized, Role: FOLLOWER
I20260924 03:02:35.356907 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P e22f16deebe84552be594e3177fe97b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22f16deebe84552be594e3177fe97b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46523 } }
I20260924 03:02:35.356990 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P e22f16deebe84552be594e3177fe97b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:02:35.357043 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P e22f16deebe84552be594e3177fe97b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:02:35.357096 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e22f16deebe84552be594e3177fe97b3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:35.357599 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 1475
I20260924 03:02:35.357666 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:02:35.357625 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P e22f16deebe84552be594e3177fe97b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22f16deebe84552be594e3177fe97b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46523 } }
I20260924 03:02:35.357776 1553 leader_election.cc:302] T 00000000000000000000000000000000 P e22f16deebe84552be594e3177fe97b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e22f16deebe84552be594e3177fe97b3; no voters:
I20260924 03:02:35.357832 392 external_mini_cluster.cc:1428] Setting key 045e4466efe710f909da5e77df564ea1
I20260924 03:02:35.357945 1553 leader_election.cc:288] T 00000000000000000000000000000000 P e22f16deebe84552be594e3177fe97b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:02:35.358011 1556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e22f16deebe84552be594e3177fe97b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:35.358194 1556 raft_consensus.cc:686] T 00000000000000000000000000000000 P e22f16deebe84552be594e3177fe97b3 [term 1 LEADER]: Becoming Leader. State: Replica: e22f16deebe84552be594e3177fe97b3, State: Running, Role: LEADER
I20260924 03:02:35.358230 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P e22f16deebe84552be594e3177fe97b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:02:35.359280 1556 consensus_queue.cc:237] T 00000000000000000000000000000000 P e22f16deebe84552be594e3177fe97b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22f16deebe84552be594e3177fe97b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46523 } }
I20260924 03:02:35.360086 1558 sys_catalog.cc:455] T 00000000000000000000000000000000 P e22f16deebe84552be594e3177fe97b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e22f16deebe84552be594e3177fe97b3. Latest consensus state: current_term: 1 leader_uuid: "e22f16deebe84552be594e3177fe97b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22f16deebe84552be594e3177fe97b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46523 } } }
I20260924 03:02:35.360181 1558 sys_catalog.cc:458] T 00000000000000000000000000000000 P e22f16deebe84552be594e3177fe97b3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:35.360385 1572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:02:35.360954 1572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:02:35.361438 1557 sys_catalog.cc:455] T 00000000000000000000000000000000 P e22f16deebe84552be594e3177fe97b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e22f16deebe84552be594e3177fe97b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22f16deebe84552be594e3177fe97b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46523 } } }
I20260924 03:02:35.361506 1557 sys_catalog.cc:458] T 00000000000000000000000000000000 P e22f16deebe84552be594e3177fe97b3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:35.362447 1572 catalog_manager.cc:1347] Generated new cluster ID: 2203aae926404eea8427d56ce57b4181
I20260924 03:02:35.362496 1572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:02:35 dist-test-slave-wgsb krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:35.369546 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:35.358368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53588 (local address 127.0.98.62:46523)
0924 03:02:35.358671 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:35.358680 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:35.358704 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:02:35.358843 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:35.358850 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:35.358932 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:35.359072 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:35.359944 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.361279 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.362333 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.362958 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.363071 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:35.365884 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:35.365888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:35.365888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:35.365900 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:35.367911 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:35.368315 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:35.368318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:35.368318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:35.368429 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:35.368638 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:35.368639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:35.368639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:35.368927 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:35.369066 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:35.369189 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 03:02:35.369319 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":65,"threads_started":1}
I20260924 03:02:35.372983 1572 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:02:35.373467 1572 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:02:35.376009 1572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e22f16deebe84552be594e3177fe97b3: Generated new TSK 0
I20260924 03:02:35.376192 1572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:02:35.404906 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46523
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:35.488018 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:35.488265 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:35.488310 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:35.489509 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:35.489585 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:35.489621 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:02:35.489653 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:02:35.491291 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:35.491586 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:35.491856 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:35.497051 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:35.497108 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:35.497138 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:35.497170 1579 server_base.cc:1034] running on GCE node
I20260924 03:02:35.497491 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:35.497766 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:35.498940 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790218955498923 us; error 36 us; skew 500 ppm
Sep 24 03:02:35 dist-test-slave-wgsb krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:35.501556 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:02:35.501987 1579 webserver.cc:460] Webserver started at http://127.0.98.1:33179/ using document root <none> and password file <none>
I20260924 03:02:35.502218 1579 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:35.502264 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:35.502408 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:35.503484 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-0/data/instance:
uuid: "22063e40d22d4fceb578e8f126d3bcd4"
format_stamp: "Formatted at 2026-09-24 03:02:35 on dist-test-slave-wgsb"
server_key: "c2081664ee3391a25c2be546de5672d2"
server_key_iv: "bbf644dcb67934c8742d1f60b8e9fe17"
server_key_version: "encryptionkey@0"
I20260924 03:02:35.503916 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-0/wal/instance:
uuid: "22063e40d22d4fceb578e8f126d3bcd4"
format_stamp: "Formatted at 2026-09-24 03:02:35 on dist-test-slave-wgsb"
server_key: "c2081664ee3391a25c2be546de5672d2"
server_key_iv: "bbf644dcb67934c8742d1f60b8e9fe17"
server_key_version: "encryptionkey@0"
I20260924 03:02:35.505281 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:35.506145 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:35.506351 1579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:35.506425 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "22063e40d22d4fceb578e8f126d3bcd4"
format_stamp: "Formatted at 2026-09-24 03:02:35 on dist-test-slave-wgsb"
server_key: "c2081664ee3391a25c2be546de5672d2"
server_key_iv: "bbf644dcb67934c8742d1f60b8e9fe17"
server_key_version: "encryptionkey@0"
I20260924 03:02:35.506537 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:35.523907 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:35.526261 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:35.526407 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:35.526968 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:35.527005 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:35.527029 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:35.527041 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:35.534130 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46425
I20260924 03:02:35.534266 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46425 every 8 connection(s)
I20260924 03:02:35.534544 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 03:02:35 dist-test-slave-wgsb krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:35.539053 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:35.527732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48673 (local address 127.0.98.62:46523)
0924 03:02:35.527851 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:35.527854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:35.528049 (+ 195us) server_negotiation.cc:407] Connection header received
0924 03:02:35.528817 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:35.528819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:35.528833 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:35.528911 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:35.529848 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.530808 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.532549 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.533059 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.533146 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:35.535986 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:35.535988 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:35.535988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:35.535998 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:35.537829 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:35.538206 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:35.538207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:35.538207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:35.538270 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:35.538517 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:35.538518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:35.538518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:35.538640 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:35.538701 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:35.538915 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 03:02:35.538988 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 03:02:35.539275 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:35.527723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46523 (local address 127.0.98.1:48673)
0924 03:02:35.527907 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:35.527955 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:35.528683 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:35.529002 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:35.529007 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:35.529420 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:35.529748 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:35.529756 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.530924 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.530925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:35.532427 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:35.532443 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.533181 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.533182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:35.533380 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:35.534007 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:35.534018 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:35.535886 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:35.537910 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:35.537916 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:35.537918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:35.538131 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:35.538325 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:35.538325 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:35.538326 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:35.538418 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:35.538700 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:35.538704 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:35.538816 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 03:02:35.538888 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 03:02:35.539023 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":72,"threads_started":1}
I20260924 03:02:35.540584 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 1579
I20260924 03:02:35.540693 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:02:35.540779 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46523
I20260924 03:02:35.540876 1710 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:35.540879 392 external_mini_cluster.cc:1428] Setting key e8223c4ec419bb887601cf6cf47c58f8
I20260924 03:02:35.541076 1710 heartbeater.cc:511] Master 127.0.98.62:46523 requested a full tablet report, sending...
I20260924 03:02:35.541658 1507 ts_manager.cc:194] Registered new tserver with Master: 22063e40d22d4fceb578e8f126d3bcd4 (127.0.98.1:46425)
I20260924 03:02:35.542577 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48673
I20260924 03:02:35.547395 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:35.540828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35319 (local address 127.0.98.62:46523)
0924 03:02:35.540871 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:35.540874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:35.541083 (+ 209us) server_negotiation.cc:407] Connection header received
0924 03:02:35.541177 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:35.541178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:35.541191 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:35.541221 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:35.541662 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.542466 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.543372 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.543577 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.543683 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:35.544737 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:35.544738 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:35.544739 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:35.544746 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:35.546255 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:35.546602 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:35.546603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:35.546604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:35.546662 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:35.546859 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:35.546860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:35.546861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:35.546980 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:35.547055 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:35.547217 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 03:02:35.547299 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":197,"server-negotiator.queue_time_us":21}
I20260924 03:02:35.547406 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:35.540826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46523 (local address 127.0.98.1:35319)
0924 03:02:35.541022 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:35.541033 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:35.541107 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:35.541282 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:35.541283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:35.541484 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:35.541558 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:35.541560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.542564 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.542565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:35.543270 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:35.543272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.543683 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.543684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:35.543749 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:35.544294 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:35.544298 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:35.544630 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:35.546350 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:35.546351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:35.546352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:35.546530 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:35.546728 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:35.546729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:35.546729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:35.546787 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:35.547047 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:35.547047 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:35.547104 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 03:02:35.547192 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 03:02:35.547298 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":118,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:02:35.570747 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46523
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:35.651436 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:35.651708 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:35.651751 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:35.652943 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:35.653020 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:35.653054 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:02:35.653096 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:02:35.654663 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:35.654940 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:35.655175 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:35.660316 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:35.660591 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:35.660594 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:35.660759 1719 server_base.cc:1034] running on GCE node
I20260924 03:02:35.660976 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:35.661233 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:35.662410 1719 hybrid_clock.cc:648] HybridClock initialized: now 1790218955662391 us; error 34 us; skew 500 ppm
Sep 24 03:02:35 dist-test-slave-wgsb krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:35.665231 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:02:35.665705 1719 webserver.cc:460] Webserver started at http://127.0.98.2:33749/ using document root <none> and password file <none>
I20260924 03:02:35.665946 1719 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:35.666013 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:35.666150 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:35.667236 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-1/data/instance:
uuid: "c850de68a47b4845aa0cca514af1fb5c"
format_stamp: "Formatted at 2026-09-24 03:02:35 on dist-test-slave-wgsb"
server_key: "b0ecd1cd23a89d3e5f148fe1a1bc435a"
server_key_iv: "841554c734fff72d31b7f13eb19fffde"
server_key_version: "encryptionkey@0"
I20260924 03:02:35.667675 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c850de68a47b4845aa0cca514af1fb5c"
format_stamp: "Formatted at 2026-09-24 03:02:35 on dist-test-slave-wgsb"
server_key: "b0ecd1cd23a89d3e5f148fe1a1bc435a"
server_key_iv: "841554c734fff72d31b7f13eb19fffde"
server_key_version: "encryptionkey@0"
I20260924 03:02:35.669049 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:02:35.669812 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:35.670064 1719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:02:35.670168 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "c850de68a47b4845aa0cca514af1fb5c"
format_stamp: "Formatted at 2026-09-24 03:02:35 on dist-test-slave-wgsb"
server_key: "b0ecd1cd23a89d3e5f148fe1a1bc435a"
server_key_iv: "841554c734fff72d31b7f13eb19fffde"
server_key_version: "encryptionkey@0"
I20260924 03:02:35.670257 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:35.703940 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:35.706310 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:35.706470 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:35.707037 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:35.707095 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:35.707137 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:35.707166 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:02:35 dist-test-slave-wgsb krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:35.714337 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41951
I20260924 03:02:35.714459 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41951 every 8 connection(s)
I20260924 03:02:35.714730 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:02:35.715854 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 1719
I20260924 03:02:35.715955 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:02:35.716123 392 external_mini_cluster.cc:1428] Setting key 9ac6fbe70982b714753ea5cb8b966970
I20260924 03:02:35.718150 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:35.707752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59005 (local address 127.0.98.62:46523)
0924 03:02:35.707864 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:35.707866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:35.708048 (+ 182us) server_negotiation.cc:407] Connection header received
0924 03:02:35.708750 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:35.708752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:35.708769 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:35.708847 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:35.709583 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.710456 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.711541 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.711903 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.711978 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:35.714906 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:35.714908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:35.714908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:35.714917 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:35.716578 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:35.717083 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:35.717084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:35.717084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:35.717135 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:35.717425 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:35.717426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:35.717426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:35.717558 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:35.717615 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:35.717938 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 03:02:35.718015 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 03:02:35.718369 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:35.707758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46523 (local address 127.0.98.2:59005)
0924 03:02:35.707931 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:35.707967 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:35.708544 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:35.708933 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:35.708938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:35.709247 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:35.709474 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:35.709479 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.710595 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.710596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:35.711433 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:35.711435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.712016 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.712017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:35.712196 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:35.713141 (+ 945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:35.713150 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:35.714823 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:35.716659 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:35.716664 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:35.716666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:35.717006 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:35.717216 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:35.717217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:35.717217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:35.717361 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:35.717623 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:35.717628 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:35.717797 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 03:02:35.717913 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 03:02:35.718098 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
I20260924 03:02:35.719012 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46523
I20260924 03:02:35.719091 1850 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:35.719280 1850 heartbeater.cc:511] Master 127.0.98.62:46523 requested a full tablet report, sending...
I20260924 03:02:35.719677 1507 ts_manager.cc:194] Registered new tserver with Master: c850de68a47b4845aa0cca514af1fb5c (127.0.98.2:41951)
I20260924 03:02:35.720155 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59005
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 03:02:35.726015 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:35.720116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45781 (local address 127.0.98.62:46523)
0924 03:02:35.720161 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:35.720162 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:35.720274 (+ 112us) server_negotiation.cc:407] Connection header received
0924 03:02:35.720385 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:35.720385 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:35.720395 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:35.720440 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:35.720760 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.721296 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.722016 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.722221 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.722312 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:35.723251 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:35.723252 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:35.723253 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:35.723259 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:35.724779 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:35.725164 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:35.725166 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:35.725166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:35.725226 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:35.725425 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:35.725426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:35.725427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:35.725573 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:35.725665 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:35.725842 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 03:02:35.725917 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:02:35.726029 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:35.720049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46523 (local address 127.0.98.2:45781)
0924 03:02:35.720215 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:35.720225 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:35.720298 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:35.720443 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:35.720444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:35.720627 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:35.720675 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:35.720676 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.721393 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.721394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:35.721919 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:35.721920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.722314 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.722315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:35.722379 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:35.722854 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:35.722859 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:35.723143 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:35.724904 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:35.724905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:35.724906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:35.725089 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:35.725291 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:35.725292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:35.725293 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:35.725351 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:35.725659 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:35.725660 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:35.725712 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 03:02:35.725815 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 03:02:35.725917 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":63,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:02:35.743283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46523
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:35.824442 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:35.824698 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:35.824739 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:35.825963 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:35.826025 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:35.826061 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:02:35.826098 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:02:35.827574 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:35.827888 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:35.828133 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:35.833397 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:35.833500 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:35.833633 1859 server_base.cc:1034] running on GCE node
W20260924 03:02:35.833397 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:35.833915 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:35.834154 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:35.835312 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790218955835303 us; error 38 us; skew 500 ppm
Sep 24 03:02:35 dist-test-slave-wgsb krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:35.837910 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:02:35.838498 1859 webserver.cc:460] Webserver started at http://127.0.98.3:41927/ using document root <none> and password file <none>
I20260924 03:02:35.838729 1859 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:35.838795 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:35.838953 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:35.840191 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-2/data/instance:
uuid: "ddf11771925343bf872e404886984b3c"
format_stamp: "Formatted at 2026-09-24 03:02:35 on dist-test-slave-wgsb"
server_key: "f2283850be63c6e5a7bcc2d48b23ca0a"
server_key_iv: "d6b0ef8512c1dc4acd68c9c168d43950"
server_key_version: "encryptionkey@0"
I20260924 03:02:35.840561 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ddf11771925343bf872e404886984b3c"
format_stamp: "Formatted at 2026-09-24 03:02:35 on dist-test-slave-wgsb"
server_key: "f2283850be63c6e5a7bcc2d48b23ca0a"
server_key_iv: "d6b0ef8512c1dc4acd68c9c168d43950"
server_key_version: "encryptionkey@0"
I20260924 03:02:35.842048 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 03:02:35.842811 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:35.843019 1859 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:35.843114 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "ddf11771925343bf872e404886984b3c"
format_stamp: "Formatted at 2026-09-24 03:02:35 on dist-test-slave-wgsb"
server_key: "f2283850be63c6e5a7bcc2d48b23ca0a"
server_key_iv: "d6b0ef8512c1dc4acd68c9c168d43950"
server_key_version: "encryptionkey@0"
I20260924 03:02:35.843199 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:35.876427 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:35.878731 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:35.878895 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:35.879396 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:35.879462 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:35.879511 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:35.879536 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:02:35 dist-test-slave-wgsb krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:35.886708 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41231
I20260924 03:02:35.886813 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41231 every 8 connection(s)
I20260924 03:02:35.887117 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:02:35.888475 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 1859
I20260924 03:02:35.888547 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:02:35.888727 392 external_mini_cluster.cc:1428] Setting key d802127a9449eccf8d96e8fea109e020
I20260924 03:02:35.889700 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:35.880130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53861 (local address 127.0.98.62:46523)
0924 03:02:35.880181 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:35.880183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:35.880326 (+ 143us) server_negotiation.cc:407] Connection header received
0924 03:02:35.880867 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:35.880869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:35.880881 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:35.880932 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:35.881594 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.882407 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.883452 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.883753 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.883842 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:35.886513 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:35.886515 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:35.886516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:35.886525 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:35.888131 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:35.888667 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:35.888668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:35.888668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:35.888732 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:35.889007 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:35.889009 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:35.889009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:35.889135 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:35.889218 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:35.889510 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 03:02:35.889595 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 03:02:35.889930 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:35.879977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46523 (local address 127.0.98.3:53861)
0924 03:02:35.880191 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:35.880243 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:35.880748 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:35.881006 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:35.881010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:35.881302 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:35.881487 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:35.881492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.882495 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.882496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:35.883352 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:35.883354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.883866 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.883867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:35.884013 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:35.884543 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:35.884555 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:35.886406 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:35.888242 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:35.888249 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:35.888251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:35.888588 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:35.888808 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:35.888809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:35.888810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:35.888931 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:35.889209 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:35.889215 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:35.889368 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 03:02:35.889487 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 03:02:35.889674 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":78,"threads_started":1}
I20260924 03:02:35.890426 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46523
I20260924 03:02:35.890517 1990 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:35.890694 1990 heartbeater.cc:511] Master 127.0.98.62:46523 requested a full tablet report, sending...
I20260924 03:02:35.891067 1506 ts_manager.cc:194] Registered new tserver with Master: ddf11771925343bf872e404886984b3c (127.0.98.3:41231)
I20260924 03:02:35.891537 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53861
I20260924 03:02:35.896914 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:35.891565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38107 (local address 127.0.98.62:46523)
0924 03:02:35.891628 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:35.891630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:35.891696 (+ 66us) server_negotiation.cc:407] Connection header received
0924 03:02:35.891773 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:35.891774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:35.891782 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:35.891842 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:35.892206 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.892646 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.893352 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.893565 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.893614 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:35.894554 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:35.894555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:35.894555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:35.894559 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:35.895852 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:35.896175 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:35.896176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:35.896176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:35.896224 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:35.896435 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:35.896436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:35.896436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:35.896525 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:35.896573 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:35.896798 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 03:02:35.896855 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:02:35.896958 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:35.891474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46523 (local address 127.0.98.3:38107)
0924 03:02:35.891634 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:35.891646 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:35.891710 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:35.891851 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:35.891852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:35.892055 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:35.892131 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:35.892133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.892745 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.892746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:35.893252 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:35.893253 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.893652 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.893653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:35.893736 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:35.894193 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:35.894198 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:35.894486 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:35.895937 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:35.895938 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:35.895938 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:35.896110 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:35.896312 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:35.896313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:35.896313 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:35.896372 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:35.896631 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:35.896632 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:35.896681 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 03:02:35.896754 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 03:02:35.896833 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
I20260924 03:02:35.900334 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 03:02:35 dist-test-slave-wgsb krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218955, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 03:02:35 dist-test-slave-wgsb krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218955, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:35.934144 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:35.925670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53600 (local address 127.0.98.62:46523)
0924 03:02:35.925741 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:35.925755 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:35.925766 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:02:35.925799 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:35.925800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:35.925812 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:35.925892 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:35.926797 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.927559 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.928467 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.928764 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.928845 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:35.931363 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:35.931365 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:35.931365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:35.931374 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:35.932876 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:35.933315 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:35.933317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:35.933317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:35.933390 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:35.933599 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:35.933600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:35.933600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:35.933718 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:35.933824 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:35.933955 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 03:02:35.934034 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 03:02:35.935844 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:02:35.937361 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:02:35.943986 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:35.940080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41231 (local address 127.0.0.1:41464)
0924 03:02:35.940419 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:35.940428 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:35.940541 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:35.940755 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:35.940756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:35.940757 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:35.940794 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:35.940796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.941628 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.941629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:35.942812 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:35.942814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.943655 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.943656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:35.943734 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:35.943749 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 03:02:35.943846 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 03:02:35.943901 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":74,"threads_started":1}
I20260924 03:02:35.944005 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:35.940280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41464 (local address 127.0.98.3:41231)
0924 03:02:35.940519 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:35.940524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:35.940536 (+ 12us) server_negotiation.cc:407] Connection header received
0924 03:02:35.940609 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:35.940611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:35.940626 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:35.940698 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:35.940850 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.941541 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.942909 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.943533 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.943636 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:35.943718 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:35.943866 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 03:02:35.943907 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":109,"threads_started":1}
I20260924 03:02:35.944172 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:35.939807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46425 (local address 127.0.0.1:42754)
0924 03:02:35.940043 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:35.940090 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:35.940157 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:35.940682 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:35.940683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:35.940685 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:35.940740 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:35.940742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.942062 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.942064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:35.943001 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:35.943003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.943934 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.943936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:35.944012 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:35.944017 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:02:35.944048 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 03:02:35.944092 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":60,"threads_started":1}
I20260924 03:02:35.944389 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:35.939984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35272 (local address 127.0.98.2:41951)
0924 03:02:35.940241 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:35.940247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:35.940260 (+ 13us) server_negotiation.cc:407] Connection header received
0924 03:02:35.940493 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:35.940496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:35.940514 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:35.940639 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:35.940818 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.941947 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.943205 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.943898 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.944137 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:35.944219 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:35.944246 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 03:02:35.944295 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":83,"threads_started":1}
I20260924 03:02:35.944588 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:35.939866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41951 (local address 127.0.0.1:35272)
0924 03:02:35.940147 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:35.940156 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:35.940325 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:35.940650 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:35.940654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:35.940656 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:35.940736 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:35.940738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.942038 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.942040 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:35.943131 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:35.943133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.943979 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.943980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:35.944032 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:35.944035 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:02:35.944064 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 03:02:35.944095 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":24,"thread_start_us":106,"threads_started":1}
I20260924 03:02:35.944698 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:35.940011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42754 (local address 127.0.98.1:46425)
0924 03:02:35.940519 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:35.940524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:35.940533 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:02:35.940571 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:35.940573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:35.940588 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:35.940836 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:35.940901 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.941956 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.943093 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:35.943826 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:35.944295 (+ 469us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:35.944391 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:35.944417 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 03:02:35.944588 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":76,"threads_started":1}
I20260924 03:02:35.945180 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 4596ff278ffa49749f0f8016fed36798 (DEFAULT_TABLE table=test-table [id=1db696f903ee4bf1a84b695e5691236d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:35.945245 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 4596ff278ffa49749f0f8016fed36798 (DEFAULT_TABLE table=test-table [id=1db696f903ee4bf1a84b695e5691236d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:35.945549 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4596ff278ffa49749f0f8016fed36798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:35.945578 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4596ff278ffa49749f0f8016fed36798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:35.948248 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 4596ff278ffa49749f0f8016fed36798 (DEFAULT_TABLE table=test-table [id=1db696f903ee4bf1a84b695e5691236d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:35.948613 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4596ff278ffa49749f0f8016fed36798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:35.948817 2013 tablet_bootstrap.cc:492] T 4596ff278ffa49749f0f8016fed36798 P ddf11771925343bf872e404886984b3c: Bootstrap starting.
I20260924 03:02:35.949007 2014 tablet_bootstrap.cc:492] T 4596ff278ffa49749f0f8016fed36798 P c850de68a47b4845aa0cca514af1fb5c: Bootstrap starting.
I20260924 03:02:35.949430 2013 tablet_bootstrap.cc:654] T 4596ff278ffa49749f0f8016fed36798 P ddf11771925343bf872e404886984b3c: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:35.949587 2014 tablet_bootstrap.cc:654] T 4596ff278ffa49749f0f8016fed36798 P c850de68a47b4845aa0cca514af1fb5c: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:35.949781 2013 log.cc:824] T 4596ff278ffa49749f0f8016fed36798 P ddf11771925343bf872e404886984b3c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:35.949894 2014 log.cc:824] T 4596ff278ffa49749f0f8016fed36798 P c850de68a47b4845aa0cca514af1fb5c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:35.950609 2014 tablet_bootstrap.cc:492] T 4596ff278ffa49749f0f8016fed36798 P c850de68a47b4845aa0cca514af1fb5c: No bootstrap required, opened a new log
I20260924 03:02:35.950640 2013 tablet_bootstrap.cc:492] T 4596ff278ffa49749f0f8016fed36798 P ddf11771925343bf872e404886984b3c: No bootstrap required, opened a new log
I20260924 03:02:35.950695 2014 ts_tablet_manager.cc:1397] T 4596ff278ffa49749f0f8016fed36798 P c850de68a47b4845aa0cca514af1fb5c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:02:35.950699 2013 ts_tablet_manager.cc:1397] T 4596ff278ffa49749f0f8016fed36798 P ddf11771925343bf872e404886984b3c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:02:35.952315 2014 raft_consensus.cc:348] T 4596ff278ffa49749f0f8016fed36798 P c850de68a47b4845aa0cca514af1fb5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } }
I20260924 03:02:35.952472 2014 raft_consensus.cc:374] T 4596ff278ffa49749f0f8016fed36798 P c850de68a47b4845aa0cca514af1fb5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:35.952515 2014 raft_consensus.cc:729] T 4596ff278ffa49749f0f8016fed36798 P c850de68a47b4845aa0cca514af1fb5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c850de68a47b4845aa0cca514af1fb5c, State: Initialized, Role: FOLLOWER
I20260924 03:02:35.952625 2014 consensus_queue.cc:260] T 4596ff278ffa49749f0f8016fed36798 P c850de68a47b4845aa0cca514af1fb5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } }
I20260924 03:02:35.952828 2017 tablet_bootstrap.cc:492] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4: Bootstrap starting.
I20260924 03:02:35.952924 2014 ts_tablet_manager.cc:1428] T 4596ff278ffa49749f0f8016fed36798 P c850de68a47b4845aa0cca514af1fb5c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:02:35.952821 2013 raft_consensus.cc:348] T 4596ff278ffa49749f0f8016fed36798 P ddf11771925343bf872e404886984b3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } }
I20260924 03:02:35.953006 2013 raft_consensus.cc:374] T 4596ff278ffa49749f0f8016fed36798 P ddf11771925343bf872e404886984b3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:35.953015 1850 heartbeater.cc:503] Master 127.0.98.62:46523 was elected leader, sending a full tablet report...
I20260924 03:02:35.953058 2013 raft_consensus.cc:729] T 4596ff278ffa49749f0f8016fed36798 P ddf11771925343bf872e404886984b3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddf11771925343bf872e404886984b3c, State: Initialized, Role: FOLLOWER
I20260924 03:02:35.953189 2013 consensus_queue.cc:260] T 4596ff278ffa49749f0f8016fed36798 P ddf11771925343bf872e404886984b3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } }
I20260924 03:02:35.953454 2013 ts_tablet_manager.cc:1428] T 4596ff278ffa49749f0f8016fed36798 P ddf11771925343bf872e404886984b3c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:02:35.953573 1990 heartbeater.cc:503] Master 127.0.98.62:46523 was elected leader, sending a full tablet report...
I20260924 03:02:35.953794 2017 tablet_bootstrap.cc:654] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:35.954288 2017 log.cc:824] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:35.955009 2017 tablet_bootstrap.cc:492] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4: No bootstrap required, opened a new log
I20260924 03:02:35.955062 2017 ts_tablet_manager.cc:1397] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:02:35.956674 2017 raft_consensus.cc:348] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } }
I20260924 03:02:35.956796 2017 raft_consensus.cc:374] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:35.956816 2017 raft_consensus.cc:729] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22063e40d22d4fceb578e8f126d3bcd4, State: Initialized, Role: FOLLOWER
I20260924 03:02:35.956923 2017 consensus_queue.cc:260] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } }
I20260924 03:02:35.957175 2017 ts_tablet_manager.cc:1428] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:02:35.957188 1710 heartbeater.cc:503] Master 127.0.98.62:46523 was elected leader, sending a full tablet report...
W20260924 03:02:35.965344 1851 tablet.cc:2369] T 4596ff278ffa49749f0f8016fed36798 P c850de68a47b4845aa0cca514af1fb5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:02:36.035351 1711 tablet.cc:2369] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:02:36.113747 2021 raft_consensus.cc:482] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:02:36.113953 2021 raft_consensus.cc:504] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } }
I20260924 03:02:36.114374 2021 leader_election.cc:288] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c850de68a47b4845aa0cca514af1fb5c (127.0.98.2:41951), ddf11771925343bf872e404886984b3c (127.0.98.3:41231)
I20260924 03:02:36.118695 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:36.114731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32793 (local address 127.0.98.2:41951)
0924 03:02:36.114800 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:36.114802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:36.114818 (+ 16us) server_negotiation.cc:407] Connection header received
0924 03:02:36.114914 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:36.114915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:36.114949 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:36.115011 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:36.115241 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.116020 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.117739 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.118314 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.118387 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:36.118398 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:36.118569 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 03:02:36.118617 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 03:02:36.118731 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:36.114598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41951 (local address 127.0.98.1:32793)
0924 03:02:36.114735 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:36.114748 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:36.114804 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:36.115040 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:36.115041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:36.115043 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:36.115137 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:36.115138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.116100 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.116101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:36.117637 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:36.117639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.118412 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.118413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:36.118480 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:36.118484 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:02:36.118566 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 03:02:36.118620 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":101,"thread_start_us":86,"threads_started":1}
I20260924 03:02:36.118791 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:36.115031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43877 (local address 127.0.98.3:41231)
0924 03:02:36.115081 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:36.115083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:36.115095 (+ 12us) server_negotiation.cc:407] Connection header received
0924 03:02:36.115116 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:36.115117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:36.115125 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:36.115166 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:36.115366 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.116107 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.117742 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.118314 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.118387 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:36.118398 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:36.118672 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 03:02:36.118717 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 03:02:36.119088 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:36.114795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41231 (local address 127.0.98.1:43877)
0924 03:02:36.114906 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:36.114918 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:36.115018 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:36.115194 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:36.115195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:36.115198 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:36.115254 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:36.115255 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.116175 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.116176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:36.117637 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:36.117639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.118477 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.118478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:36.118592 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:36.118596 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:02:36.118654 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 03:02:36.118696 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":10,"thread_start_us":57,"threads_started":1}
I20260924 03:02:36.119249 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4596ff278ffa49749f0f8016fed36798" candidate_uuid: "22063e40d22d4fceb578e8f126d3bcd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c850de68a47b4845aa0cca514af1fb5c" is_pre_election: true
I20260924 03:02:36.119477 1804 raft_consensus.cc:2454] T 4596ff278ffa49749f0f8016fed36798 P c850de68a47b4845aa0cca514af1fb5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22063e40d22d4fceb578e8f126d3bcd4 in term 0.
I20260924 03:02:36.119428 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4596ff278ffa49749f0f8016fed36798" candidate_uuid: "22063e40d22d4fceb578e8f126d3bcd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddf11771925343bf872e404886984b3c" is_pre_election: true
I20260924 03:02:36.119632 1944 raft_consensus.cc:2454] T 4596ff278ffa49749f0f8016fed36798 P ddf11771925343bf872e404886984b3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22063e40d22d4fceb578e8f126d3bcd4 in term 0.
I20260924 03:02:36.119772 1597 leader_election.cc:302] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22063e40d22d4fceb578e8f126d3bcd4, c850de68a47b4845aa0cca514af1fb5c; no voters:
I20260924 03:02:36.120035 2021 raft_consensus.cc:2781] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:02:36.120115 2021 raft_consensus.cc:482] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:02:36.120151 2021 raft_consensus.cc:3037] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:36.120921 2021 raft_consensus.cc:504] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } }
I20260924 03:02:36.121058 2021 leader_election.cc:288] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4 [CANDIDATE]: Term 1 election: Requested vote from peers c850de68a47b4845aa0cca514af1fb5c (127.0.98.2:41951), ddf11771925343bf872e404886984b3c (127.0.98.3:41231)
I20260924 03:02:36.121228 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4596ff278ffa49749f0f8016fed36798" candidate_uuid: "22063e40d22d4fceb578e8f126d3bcd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c850de68a47b4845aa0cca514af1fb5c"
I20260924 03:02:36.121315 1804 raft_consensus.cc:3037] T 4596ff278ffa49749f0f8016fed36798 P c850de68a47b4845aa0cca514af1fb5c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:36.121333 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4596ff278ffa49749f0f8016fed36798" candidate_uuid: "22063e40d22d4fceb578e8f126d3bcd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddf11771925343bf872e404886984b3c"
I20260924 03:02:36.121423 1944 raft_consensus.cc:3037] T 4596ff278ffa49749f0f8016fed36798 P ddf11771925343bf872e404886984b3c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:36.122061 1804 raft_consensus.cc:2454] T 4596ff278ffa49749f0f8016fed36798 P c850de68a47b4845aa0cca514af1fb5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22063e40d22d4fceb578e8f126d3bcd4 in term 1.
I20260924 03:02:36.122280 1597 leader_election.cc:302] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22063e40d22d4fceb578e8f126d3bcd4, c850de68a47b4845aa0cca514af1fb5c; no voters:
I20260924 03:02:36.122315 1944 raft_consensus.cc:2454] T 4596ff278ffa49749f0f8016fed36798 P ddf11771925343bf872e404886984b3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22063e40d22d4fceb578e8f126d3bcd4 in term 1.
I20260924 03:02:36.122424 2021 raft_consensus.cc:2781] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:36.122536 2021 raft_consensus.cc:686] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4 [term 1 LEADER]: Becoming Leader. State: Replica: 22063e40d22d4fceb578e8f126d3bcd4, State: Running, Role: LEADER
I20260924 03:02:36.122642 2021 consensus_queue.cc:237] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } }
I20260924 03:02:36.123669 1506 catalog_manager.cc:5495] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22063e40d22d4fceb578e8f126d3bcd4 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "22063e40d22d4fceb578e8f126d3bcd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } health_report { overall_health: UNKNOWN } } }
W20260924 03:02:36.138679 1991 tablet.cc:2369] T 4596ff278ffa49749f0f8016fed36798 P ddf11771925343bf872e404886984b3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:02:36.171221 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:36.168459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42764 (local address 127.0.98.1:46425)
0924 03:02:36.168553 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:36.168555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:36.168564 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:02:36.168639 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:36.168639 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:36.168658 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:36.168718 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:02:36.168877 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.169597 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.170392 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.170576 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.170642 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:36.170857 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:02:36.170957 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:02:36.171017 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:36.171083 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 03:02:36.171101 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 03:02:36.173640 1944 raft_consensus.cc:1261] T 4596ff278ffa49749f0f8016fed36798 P ddf11771925343bf872e404886984b3c [term 1 FOLLOWER]: Refusing update from remote peer 22063e40d22d4fceb578e8f126d3bcd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:02:36.173667 1804 raft_consensus.cc:1261] T 4596ff278ffa49749f0f8016fed36798 P c850de68a47b4845aa0cca514af1fb5c [term 1 FOLLOWER]: Refusing update from remote peer 22063e40d22d4fceb578e8f126d3bcd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:02:36.174064 2024 consensus_queue.cc:1035] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:02:36.174214 2021 consensus_queue.cc:1035] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:02:36.176919 2029 mvcc.cc:205] Tried to move back new op lower bound from 7332736844484390912 to 7332736844279242752. Current Snapshot: MvccSnapshot[applied={T|T < 7332736844484390912}]
I20260924 03:02:36.176625 2027 mvcc.cc:205] Tried to move back new op lower bound from 7332736844484390912 to 7332736844279242752. Current Snapshot: MvccSnapshot[applied={T|T < 7332736844484390912}]
I20260924 03:02:36.180891 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:02:36.181001 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:02:36.181797 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e22f16deebe84552be594e3177fe97b3: Sending DeleteTablet for 3 replicas of tablet 4596ff278ffa49749f0f8016fed36798
I20260924 03:02:36.182134 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 4596ff278ffa49749f0f8016fed36798 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:02:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35272
I20260924 03:02:36.182144 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 4596ff278ffa49749f0f8016fed36798 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:02:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42754
I20260924 03:02:36.182188 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 4596ff278ffa49749f0f8016fed36798 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:02:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41464
I20260924 03:02:36.182371 2039 tablet_replica.cc:330] T 4596ff278ffa49749f0f8016fed36798 P ddf11771925343bf872e404886984b3c: stopping tablet replica
I20260924 03:02:36.182390 2038 tablet_replica.cc:330] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4: stopping tablet replica
I20260924 03:02:36.182476 2039 raft_consensus.cc:2229] T 4596ff278ffa49749f0f8016fed36798 P ddf11771925343bf872e404886984b3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:02:36.182479 2038 raft_consensus.cc:2229] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:02:36.182572 2039 raft_consensus.cc:2258] T 4596ff278ffa49749f0f8016fed36798 P ddf11771925343bf872e404886984b3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:02:36.182598 2038 raft_consensus.cc:2258] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:02:36.182832 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:02:36.183018 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:02:36.183035 2039 ts_tablet_manager.cc:1905] T 4596ff278ffa49749f0f8016fed36798 P ddf11771925343bf872e404886984b3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:02:36.183043 2038 ts_tablet_manager.cc:1905] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:02:36.184448 1644 tablet_service.cc:1467] Processing CreateTablet for tablet ff9c399265564f10a3519926e9f58411 (DEFAULT_TABLE table=test-table [id=856f5b1c0b3a44e3804e3eb6e8829bc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:36.184577 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9c399265564f10a3519926e9f58411. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:36.185032 2038 ts_tablet_manager.cc:1918] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:02:36.185101 2038 log.cc:1196] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-0/wal/wals/4596ff278ffa49749f0f8016fed36798
I20260924 03:02:36.185220 2039 ts_tablet_manager.cc:1918] T 4596ff278ffa49749f0f8016fed36798 P ddf11771925343bf872e404886984b3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:02:36.185283 2039 log.cc:1196] T 4596ff278ffa49749f0f8016fed36798 P ddf11771925343bf872e404886984b3c: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-2/wal/wals/4596ff278ffa49749f0f8016fed36798
I20260924 03:02:36.185400 2038 ts_tablet_manager.cc:1939] T 4596ff278ffa49749f0f8016fed36798 P 22063e40d22d4fceb578e8f126d3bcd4: Deleting consensus metadata
I20260924 03:02:36.185554 2039 ts_tablet_manager.cc:1939] T 4596ff278ffa49749f0f8016fed36798 P ddf11771925343bf872e404886984b3c: Deleting consensus metadata
I20260924 03:02:36.186005 2017 tablet_bootstrap.cc:492] T ff9c399265564f10a3519926e9f58411 P 22063e40d22d4fceb578e8f126d3bcd4: Bootstrap starting.
I20260924 03:02:36.186066 1492 catalog_manager.cc:4841] TS 22063e40d22d4fceb578e8f126d3bcd4 (127.0.98.1:46425): tablet 4596ff278ffa49749f0f8016fed36798 (table test-table [id=1db696f903ee4bf1a84b695e5691236d]) successfully deleted
I20260924 03:02:36.186331 2017 tablet_bootstrap.cc:654] T ff9c399265564f10a3519926e9f58411 P 22063e40d22d4fceb578e8f126d3bcd4: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:36.186797 1493 catalog_manager.cc:4841] TS ddf11771925343bf872e404886984b3c (127.0.98.3:41231): tablet 4596ff278ffa49749f0f8016fed36798 (table test-table [id=1db696f903ee4bf1a84b695e5691236d]) successfully deleted
I20260924 03:02:36.186888 1784 tablet_service.cc:1467] Processing CreateTablet for tablet ff9c399265564f10a3519926e9f58411 (DEFAULT_TABLE table=test-table [id=856f5b1c0b3a44e3804e3eb6e8829bc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:36.186977 1924 tablet_service.cc:1467] Processing CreateTablet for tablet ff9c399265564f10a3519926e9f58411 (DEFAULT_TABLE table=test-table [id=856f5b1c0b3a44e3804e3eb6e8829bc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:36.187014 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9c399265564f10a3519926e9f58411. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:36.187069 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9c399265564f10a3519926e9f58411. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:36.187208 2017 tablet_bootstrap.cc:492] T ff9c399265564f10a3519926e9f58411 P 22063e40d22d4fceb578e8f126d3bcd4: No bootstrap required, opened a new log
I20260924 03:02:36.187247 2017 ts_tablet_manager.cc:1397] T ff9c399265564f10a3519926e9f58411 P 22063e40d22d4fceb578e8f126d3bcd4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:36.187377 2017 raft_consensus.cc:348] T ff9c399265564f10a3519926e9f58411 P 22063e40d22d4fceb578e8f126d3bcd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } }
I20260924 03:02:36.187446 2017 raft_consensus.cc:374] T ff9c399265564f10a3519926e9f58411 P 22063e40d22d4fceb578e8f126d3bcd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:36.187465 2017 raft_consensus.cc:729] T ff9c399265564f10a3519926e9f58411 P 22063e40d22d4fceb578e8f126d3bcd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22063e40d22d4fceb578e8f126d3bcd4, State: Initialized, Role: FOLLOWER
I20260924 03:02:36.187508 2017 consensus_queue.cc:260] T ff9c399265564f10a3519926e9f58411 P 22063e40d22d4fceb578e8f126d3bcd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } }
I20260924 03:02:36.187629 2017 ts_tablet_manager.cc:1428] T ff9c399265564f10a3519926e9f58411 P 22063e40d22d4fceb578e8f126d3bcd4: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 03:02:36.188051 2037 tablet_replica.cc:330] T 4596ff278ffa49749f0f8016fed36798 P c850de68a47b4845aa0cca514af1fb5c: stopping tablet replica
I20260924 03:02:36.188459 2014 tablet_bootstrap.cc:492] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c: Bootstrap starting.
I20260924 03:02:36.188772 2037 raft_consensus.cc:2229] T 4596ff278ffa49749f0f8016fed36798 P c850de68a47b4845aa0cca514af1fb5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:02:36.188850 2013 tablet_bootstrap.cc:492] T ff9c399265564f10a3519926e9f58411 P ddf11771925343bf872e404886984b3c: Bootstrap starting.
I20260924 03:02:36.188881 2037 raft_consensus.cc:2258] T 4596ff278ffa49749f0f8016fed36798 P c850de68a47b4845aa0cca514af1fb5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:02:36.189169 2014 tablet_bootstrap.cc:654] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:36.189675 2013 tablet_bootstrap.cc:654] T ff9c399265564f10a3519926e9f58411 P ddf11771925343bf872e404886984b3c: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:36.189819 2037 ts_tablet_manager.cc:1905] T 4596ff278ffa49749f0f8016fed36798 P c850de68a47b4845aa0cca514af1fb5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:02:36.190096 2014 tablet_bootstrap.cc:492] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c: No bootstrap required, opened a new log
I20260924 03:02:36.190141 2014 ts_tablet_manager.cc:1397] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:02:36.190302 2014 raft_consensus.cc:348] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } }
I20260924 03:02:36.190382 2014 raft_consensus.cc:374] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:36.190407 2014 raft_consensus.cc:729] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c850de68a47b4845aa0cca514af1fb5c, State: Initialized, Role: FOLLOWER
I20260924 03:02:36.190496 2013 tablet_bootstrap.cc:492] T ff9c399265564f10a3519926e9f58411 P ddf11771925343bf872e404886984b3c: No bootstrap required, opened a new log
I20260924 03:02:36.190475 2014 consensus_queue.cc:260] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } }
I20260924 03:02:36.190532 2013 ts_tablet_manager.cc:1397] T ff9c399265564f10a3519926e9f58411 P ddf11771925343bf872e404886984b3c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:02:36.190564 2014 ts_tablet_manager.cc:1428] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 03:02:36.190678 2013 raft_consensus.cc:348] T ff9c399265564f10a3519926e9f58411 P ddf11771925343bf872e404886984b3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } }
I20260924 03:02:36.190742 2013 raft_consensus.cc:374] T ff9c399265564f10a3519926e9f58411 P ddf11771925343bf872e404886984b3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:36.190764 2013 raft_consensus.cc:729] T ff9c399265564f10a3519926e9f58411 P ddf11771925343bf872e404886984b3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddf11771925343bf872e404886984b3c, State: Initialized, Role: FOLLOWER
I20260924 03:02:36.190830 2013 consensus_queue.cc:260] T ff9c399265564f10a3519926e9f58411 P ddf11771925343bf872e404886984b3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } }
I20260924 03:02:36.190917 2013 ts_tablet_manager.cc:1428] T ff9c399265564f10a3519926e9f58411 P ddf11771925343bf872e404886984b3c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 03:02:36.191411 2037 ts_tablet_manager.cc:1918] T 4596ff278ffa49749f0f8016fed36798 P c850de68a47b4845aa0cca514af1fb5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:02:36.191489 2037 log.cc:1196] T 4596ff278ffa49749f0f8016fed36798 P c850de68a47b4845aa0cca514af1fb5c: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-1/wal/wals/4596ff278ffa49749f0f8016fed36798
I20260924 03:02:36.191790 2037 ts_tablet_manager.cc:1939] T 4596ff278ffa49749f0f8016fed36798 P c850de68a47b4845aa0cca514af1fb5c: Deleting consensus metadata
I20260924 03:02:36.192342 1493 catalog_manager.cc:4841] TS c850de68a47b4845aa0cca514af1fb5c (127.0.98.2:41951): tablet 4596ff278ffa49749f0f8016fed36798 (table test-table [id=1db696f903ee4bf1a84b695e5691236d]) successfully deleted
I20260924 03:02:36.340221 2018 raft_consensus.cc:482] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:02:36.340368 2018 raft_consensus.cc:504] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } }
I20260924 03:02:36.340785 2018 leader_election.cc:288] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22063e40d22d4fceb578e8f126d3bcd4 (127.0.98.1:46425), ddf11771925343bf872e404886984b3c (127.0.98.3:41231)
I20260924 03:02:36.345180 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:36.341147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36809 (local address 127.0.98.3:41231)
0924 03:02:36.341197 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:36.341199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:36.341247 (+ 48us) server_negotiation.cc:407] Connection header received
0924 03:02:36.341314 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:36.341315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:36.341324 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:36.341378 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:36.341612 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.342454 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.344092 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.344718 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.344864 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:36.344876 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:36.345055 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 03:02:36.345101 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 03:02:36.345180 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:36.341099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38383 (local address 127.0.98.1:46425)
0924 03:02:36.341146 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:36.341148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:36.341210 (+ 62us) server_negotiation.cc:407] Connection header received
0924 03:02:36.341289 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:36.341290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:36.341303 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:36.341377 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:36.341634 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.342475 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.344078 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.344746 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.344819 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:36.344833 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:36.345059 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 03:02:36.345103 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 03:02:36.345212 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:36.340972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46425 (local address 127.0.98.2:38383)
0924 03:02:36.341129 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:36.341142 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:36.341219 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:36.341388 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:36.341389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:36.341394 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:36.341471 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:36.341473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.342584 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.342585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:36.343973 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:36.343977 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.344872 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.344872 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:36.344970 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:36.344974 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:02:36.345040 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 03:02:36.345102 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":136,"thread_start_us":68,"threads_started":1}
I20260924 03:02:36.345219 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:36.341087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41231 (local address 127.0.98.2:36809)
0924 03:02:36.341195 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:36.341204 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:36.341257 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:36.341388 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:36.341389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:36.341391 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:36.341471 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:36.341473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.342584 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.342585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:36.343973 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:36.343977 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.344870 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.344872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:36.344969 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:36.344975 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 03:02:36.345041 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 03:02:36.345102 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":74,"thread_start_us":63,"threads_started":1}
I20260924 03:02:36.345577 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9c399265564f10a3519926e9f58411" candidate_uuid: "c850de68a47b4845aa0cca514af1fb5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddf11771925343bf872e404886984b3c" is_pre_election: true
I20260924 03:02:36.345693 1944 raft_consensus.cc:2454] T ff9c399265564f10a3519926e9f58411 P ddf11771925343bf872e404886984b3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c850de68a47b4845aa0cca514af1fb5c in term 0.
I20260924 03:02:36.345896 1737 leader_election.cc:302] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c850de68a47b4845aa0cca514af1fb5c, ddf11771925343bf872e404886984b3c; no voters:
I20260924 03:02:36.346055 2018 raft_consensus.cc:2781] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:02:36.346096 2018 raft_consensus.cc:482] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:02:36.346125 2018 raft_consensus.cc:3037] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:36.346055 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9c399265564f10a3519926e9f58411" candidate_uuid: "c850de68a47b4845aa0cca514af1fb5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22063e40d22d4fceb578e8f126d3bcd4" is_pre_election: true
I20260924 03:02:36.346187 1664 raft_consensus.cc:2454] T ff9c399265564f10a3519926e9f58411 P 22063e40d22d4fceb578e8f126d3bcd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c850de68a47b4845aa0cca514af1fb5c in term 0.
I20260924 03:02:36.346926 2018 raft_consensus.cc:504] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } }
I20260924 03:02:36.347082 2018 leader_election.cc:288] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c [CANDIDATE]: Term 1 election: Requested vote from peers 22063e40d22d4fceb578e8f126d3bcd4 (127.0.98.1:46425), ddf11771925343bf872e404886984b3c (127.0.98.3:41231)
I20260924 03:02:36.347250 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9c399265564f10a3519926e9f58411" candidate_uuid: "c850de68a47b4845aa0cca514af1fb5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddf11771925343bf872e404886984b3c"
I20260924 03:02:36.347328 1944 raft_consensus.cc:3037] T ff9c399265564f10a3519926e9f58411 P ddf11771925343bf872e404886984b3c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:36.347687 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9c399265564f10a3519926e9f58411" candidate_uuid: "c850de68a47b4845aa0cca514af1fb5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22063e40d22d4fceb578e8f126d3bcd4"
I20260924 03:02:36.347754 1664 raft_consensus.cc:3037] T ff9c399265564f10a3519926e9f58411 P 22063e40d22d4fceb578e8f126d3bcd4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:36.348403 1664 raft_consensus.cc:2454] T ff9c399265564f10a3519926e9f58411 P 22063e40d22d4fceb578e8f126d3bcd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c850de68a47b4845aa0cca514af1fb5c in term 1.
I20260924 03:02:36.348598 1736 leader_election.cc:302] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22063e40d22d4fceb578e8f126d3bcd4, c850de68a47b4845aa0cca514af1fb5c; no voters:
I20260924 03:02:36.348755 2018 raft_consensus.cc:2781] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:36.348878 2018 raft_consensus.cc:686] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c [term 1 LEADER]: Becoming Leader. State: Replica: c850de68a47b4845aa0cca514af1fb5c, State: Running, Role: LEADER
I20260924 03:02:36.348953 2018 consensus_queue.cc:237] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } }
I20260924 03:02:36.349076 1944 raft_consensus.cc:2454] T ff9c399265564f10a3519926e9f58411 P ddf11771925343bf872e404886984b3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c850de68a47b4845aa0cca514af1fb5c in term 1.
I20260924 03:02:36.349687 1507 catalog_manager.cc:5495] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c reported cstate change: term changed from 0 to 1, leader changed from <none> to c850de68a47b4845aa0cca514af1fb5c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c850de68a47b4845aa0cca514af1fb5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } health_report { overall_health: UNKNOWN } } }
Sep 24 03:02:36 dist-test-slave-wgsb krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:36.395627 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:36.388435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53602 (local address 127.0.98.62:46523)
0924 03:02:36.388559 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:36.388562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:36.388572 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:02:36.388641 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:36.388642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:36.388650 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:36.388703 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:36.389184 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.389706 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.390532 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.390728 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.390790 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:36.392883 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:36.392885 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:36.392885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:36.392893 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:36.394315 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:36.394741 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:36.394742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:36.394743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:36.394797 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:36.395022 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:36.395023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:36.395023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:36.395134 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:36.395192 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:36.395399 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 03:02:36.395504 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":88}
I20260924 03:02:36.395558 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:36.388359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46523 (local address 127.0.0.1:53602)
0924 03:02:36.388512 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:36.388523 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:36.388585 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:36.388725 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:36.388726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:36.389032 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:36.389079 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:36.389082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.389783 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.389784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:36.390453 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:36.390462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.390797 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.390798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:36.390835 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:36.391396 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:36.391401 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:36.392800 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:36.394417 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:36.394425 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:36.394427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:36.394651 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:36.394884 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:36.394885 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:36.394885 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:36.394948 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:36.395217 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:36.395225 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:36.395281 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 03:02:36.395354 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 03:02:36.395455 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260924 03:02:36.398208 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53602:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 03:02:36.398454 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:02:36.400483 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 33c4f0752ba347e4904fcf691ee367ef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3ac98306e35748cbb49ef14ecd71031f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:02:36.400508 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 33c4f0752ba347e4904fcf691ee367ef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3ac98306e35748cbb49ef14ecd71031f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:02:36.400508 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 33c4f0752ba347e4904fcf691ee367ef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3ac98306e35748cbb49ef14ecd71031f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:02:36.400665 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33c4f0752ba347e4904fcf691ee367ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:36.400665 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33c4f0752ba347e4904fcf691ee367ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:36.400666 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33c4f0752ba347e4904fcf691ee367ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:36.401885 2017 tablet_bootstrap.cc:492] T 33c4f0752ba347e4904fcf691ee367ef P 22063e40d22d4fceb578e8f126d3bcd4: Bootstrap starting.
I20260924 03:02:36.402225 2013 tablet_bootstrap.cc:492] T 33c4f0752ba347e4904fcf691ee367ef P ddf11771925343bf872e404886984b3c: Bootstrap starting.
I20260924 03:02:36.402245 2014 tablet_bootstrap.cc:492] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c: Bootstrap starting.
I20260924 03:02:36.402281 2017 tablet_bootstrap.cc:654] T 33c4f0752ba347e4904fcf691ee367ef P 22063e40d22d4fceb578e8f126d3bcd4: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:36.402655 2013 tablet_bootstrap.cc:654] T 33c4f0752ba347e4904fcf691ee367ef P ddf11771925343bf872e404886984b3c: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:36.403100 2014 tablet_bootstrap.cc:654] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:36.403234 2017 tablet_bootstrap.cc:492] T 33c4f0752ba347e4904fcf691ee367ef P 22063e40d22d4fceb578e8f126d3bcd4: No bootstrap required, opened a new log
I20260924 03:02:36.403275 2017 ts_tablet_manager.cc:1397] T 33c4f0752ba347e4904fcf691ee367ef P 22063e40d22d4fceb578e8f126d3bcd4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 03:02:36.403398 2017 raft_consensus.cc:348] T 33c4f0752ba347e4904fcf691ee367ef P 22063e40d22d4fceb578e8f126d3bcd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } }
I20260924 03:02:36.403465 2017 raft_consensus.cc:374] T 33c4f0752ba347e4904fcf691ee367ef P 22063e40d22d4fceb578e8f126d3bcd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:36.403482 2017 raft_consensus.cc:729] T 33c4f0752ba347e4904fcf691ee367ef P 22063e40d22d4fceb578e8f126d3bcd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22063e40d22d4fceb578e8f126d3bcd4, State: Initialized, Role: FOLLOWER
I20260924 03:02:36.403522 2017 consensus_queue.cc:260] T 33c4f0752ba347e4904fcf691ee367ef P 22063e40d22d4fceb578e8f126d3bcd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } }
I20260924 03:02:36.403632 2017 ts_tablet_manager.cc:1428] T 33c4f0752ba347e4904fcf691ee367ef P 22063e40d22d4fceb578e8f126d3bcd4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:36.403688 2024 tablet_replica.cc:439] T 33c4f0752ba347e4904fcf691ee367ef P 22063e40d22d4fceb578e8f126d3bcd4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } } }
I20260924 03:02:36.403765 2024 tablet_replica.cc:442] T 33c4f0752ba347e4904fcf691ee367ef P 22063e40d22d4fceb578e8f126d3bcd4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:02:36.403836 2014 tablet_bootstrap.cc:492] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c: No bootstrap required, opened a new log
I20260924 03:02:36.403893 2014 ts_tablet_manager.cc:1397] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:02:36.404059 2013 tablet_bootstrap.cc:492] T 33c4f0752ba347e4904fcf691ee367ef P ddf11771925343bf872e404886984b3c: No bootstrap required, opened a new log
I20260924 03:02:36.404029 2014 raft_consensus.cc:348] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } }
I20260924 03:02:36.404089 2014 raft_consensus.cc:374] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:36.404101 2013 ts_tablet_manager.cc:1397] T 33c4f0752ba347e4904fcf691ee367ef P ddf11771925343bf872e404886984b3c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:02:36.404114 2014 raft_consensus.cc:729] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c850de68a47b4845aa0cca514af1fb5c, State: Initialized, Role: FOLLOWER
I20260924 03:02:36.404166 2014 consensus_queue.cc:260] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } }
I20260924 03:02:36.404250 2014 ts_tablet_manager.cc:1428] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:36.404242 2013 raft_consensus.cc:348] T 33c4f0752ba347e4904fcf691ee367ef P ddf11771925343bf872e404886984b3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } }
I20260924 03:02:36.404309 2013 raft_consensus.cc:374] T 33c4f0752ba347e4904fcf691ee367ef P ddf11771925343bf872e404886984b3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:36.404340 2013 raft_consensus.cc:729] T 33c4f0752ba347e4904fcf691ee367ef P ddf11771925343bf872e404886984b3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddf11771925343bf872e404886984b3c, State: Initialized, Role: FOLLOWER
I20260924 03:02:36.404361 2018 tablet_replica.cc:439] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } } }
I20260924 03:02:36.404379 2013 consensus_queue.cc:260] T 33c4f0752ba347e4904fcf691ee367ef P ddf11771925343bf872e404886984b3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } }
I20260924 03:02:36.404426 2018 tablet_replica.cc:442] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:02:36.404502 2013 ts_tablet_manager.cc:1428] T 33c4f0752ba347e4904fcf691ee367ef P ddf11771925343bf872e404886984b3c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:36.404563 2019 tablet_replica.cc:439] T 33c4f0752ba347e4904fcf691ee367ef P ddf11771925343bf872e404886984b3c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } } }
I20260924 03:02:36.404646 2019 tablet_replica.cc:442] T 33c4f0752ba347e4904fcf691ee367ef P ddf11771925343bf872e404886984b3c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:02:36.539935 2018 raft_consensus.cc:482] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:02:36.540050 2018 raft_consensus.cc:504] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } }
I20260924 03:02:36.540197 2018 leader_election.cc:288] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22063e40d22d4fceb578e8f126d3bcd4 (127.0.98.1:46425), ddf11771925343bf872e404886984b3c (127.0.98.3:41231)
I20260924 03:02:36.540382 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33c4f0752ba347e4904fcf691ee367ef" candidate_uuid: "c850de68a47b4845aa0cca514af1fb5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22063e40d22d4fceb578e8f126d3bcd4" is_pre_election: true
I20260924 03:02:36.540395 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33c4f0752ba347e4904fcf691ee367ef" candidate_uuid: "c850de68a47b4845aa0cca514af1fb5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddf11771925343bf872e404886984b3c" is_pre_election: true
I20260924 03:02:36.540496 1664 raft_consensus.cc:2454] T 33c4f0752ba347e4904fcf691ee367ef P 22063e40d22d4fceb578e8f126d3bcd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c850de68a47b4845aa0cca514af1fb5c in term 0.
I20260924 03:02:36.540493 1944 raft_consensus.cc:2454] T 33c4f0752ba347e4904fcf691ee367ef P ddf11771925343bf872e404886984b3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c850de68a47b4845aa0cca514af1fb5c in term 0.
I20260924 03:02:36.540676 1737 leader_election.cc:302] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c850de68a47b4845aa0cca514af1fb5c, ddf11771925343bf872e404886984b3c; no voters:
I20260924 03:02:36.540901 2018 raft_consensus.cc:2781] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:02:36.540966 2018 raft_consensus.cc:482] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:02:36.540995 2018 raft_consensus.cc:3037] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:36.541692 2018 raft_consensus.cc:504] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } }
I20260924 03:02:36.541853 2018 leader_election.cc:288] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c [CANDIDATE]: Term 1 election: Requested vote from peers 22063e40d22d4fceb578e8f126d3bcd4 (127.0.98.1:46425), ddf11771925343bf872e404886984b3c (127.0.98.3:41231)
I20260924 03:02:36.542011 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33c4f0752ba347e4904fcf691ee367ef" candidate_uuid: "c850de68a47b4845aa0cca514af1fb5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddf11771925343bf872e404886984b3c"
I20260924 03:02:36.542086 1944 raft_consensus.cc:3037] T 33c4f0752ba347e4904fcf691ee367ef P ddf11771925343bf872e404886984b3c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:36.542063 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33c4f0752ba347e4904fcf691ee367ef" candidate_uuid: "c850de68a47b4845aa0cca514af1fb5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22063e40d22d4fceb578e8f126d3bcd4"
I20260924 03:02:36.542178 1664 raft_consensus.cc:3037] T 33c4f0752ba347e4904fcf691ee367ef P 22063e40d22d4fceb578e8f126d3bcd4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:36.542742 1944 raft_consensus.cc:2454] T 33c4f0752ba347e4904fcf691ee367ef P ddf11771925343bf872e404886984b3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c850de68a47b4845aa0cca514af1fb5c in term 1.
I20260924 03:02:36.542826 1664 raft_consensus.cc:2454] T 33c4f0752ba347e4904fcf691ee367ef P 22063e40d22d4fceb578e8f126d3bcd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c850de68a47b4845aa0cca514af1fb5c in term 1.
I20260924 03:02:36.542937 1737 leader_election.cc:302] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c850de68a47b4845aa0cca514af1fb5c, ddf11771925343bf872e404886984b3c; no voters:
I20260924 03:02:36.543046 2018 raft_consensus.cc:2781] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:36.543133 2018 raft_consensus.cc:686] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c [term 1 LEADER]: Becoming Leader. State: Replica: c850de68a47b4845aa0cca514af1fb5c, State: Running, Role: LEADER
I20260924 03:02:36.543205 2018 consensus_queue.cc:237] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } }
I20260924 03:02:36.543530 2045 tablet_replica.cc:439] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c: TxnStatusTablet state changed. Reason: New leader c850de68a47b4845aa0cca514af1fb5c. Latest consensus state: current_term: 1 leader_uuid: "c850de68a47b4845aa0cca514af1fb5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } } }
I20260924 03:02:36.543653 2045 tablet_replica.cc:442] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:02:36.543890 1507 catalog_manager.cc:5495] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c reported cstate change: term changed from 0 to 1, leader changed from <none> to c850de68a47b4845aa0cca514af1fb5c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c850de68a47b4845aa0cca514af1fb5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } health_report { overall_health: UNKNOWN } } }
I20260924 03:02:36.544157 2057 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 03:02:36.544241 2057 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 03:02:36.554502 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:36.551692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35278 (local address 127.0.98.2:41951)
0924 03:02:36.551759 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:36.551761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:36.551788 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:02:36.551918 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:36.551919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:36.551937 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:36.552033 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:02:36.552174 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.552700 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.553604 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.553819 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.553929 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:36.554124 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:02:36.554246 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:02:36.554336 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:36.554404 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 03:02:36.554425 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 03:02:36.554541 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:36.551635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41951 (local address 127.0.0.1:35278)
0924 03:02:36.551738 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:36.551751 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:36.551811 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:36.552036 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:36.552038 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:36.552039 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:02:36.552090 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:36.552092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.552790 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.552791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:36.553489 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:36.553492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.553917 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.553918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:36.553985 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:36.554002 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:02:36.554322 (+ 320us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:02:36.554332 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 03:02:36.554398 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 03:02:36.554455 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 03:02:36.555861 1944 raft_consensus.cc:1261] T 33c4f0752ba347e4904fcf691ee367ef P ddf11771925343bf872e404886984b3c [term 1 FOLLOWER]: Refusing update from remote peer c850de68a47b4845aa0cca514af1fb5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:02:36.555920 1664 raft_consensus.cc:1261] T 33c4f0752ba347e4904fcf691ee367ef P 22063e40d22d4fceb578e8f126d3bcd4 [term 1 FOLLOWER]: Refusing update from remote peer c850de68a47b4845aa0cca514af1fb5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:02:36.556236 2045 consensus_queue.cc:1035] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:02:36.556388 2018 consensus_queue.cc:1035] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:02:36.557020 2019 tablet_replica.cc:439] T 33c4f0752ba347e4904fcf691ee367ef P ddf11771925343bf872e404886984b3c: TxnStatusTablet state changed. Reason: New leader c850de68a47b4845aa0cca514af1fb5c. Latest consensus state: current_term: 1 leader_uuid: "c850de68a47b4845aa0cca514af1fb5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } } }
I20260924 03:02:36.557117 2019 tablet_replica.cc:442] T 33c4f0752ba347e4904fcf691ee367ef P ddf11771925343bf872e404886984b3c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:02:36.557140 2024 tablet_replica.cc:439] T 33c4f0752ba347e4904fcf691ee367ef P 22063e40d22d4fceb578e8f126d3bcd4: TxnStatusTablet state changed. Reason: New leader c850de68a47b4845aa0cca514af1fb5c. Latest consensus state: current_term: 1 leader_uuid: "c850de68a47b4845aa0cca514af1fb5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } } }
I20260924 03:02:36.557236 2024 tablet_replica.cc:442] T 33c4f0752ba347e4904fcf691ee367ef P 22063e40d22d4fceb578e8f126d3bcd4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:02:36.558247 2045 tablet_replica.cc:439] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c850de68a47b4845aa0cca514af1fb5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } } }
I20260924 03:02:36.558346 2045 tablet_replica.cc:442] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:02:36.558347 2018 tablet_replica.cc:439] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c850de68a47b4845aa0cca514af1fb5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } } }
I20260924 03:02:36.558758 2018 tablet_replica.cc:442] T 33c4f0752ba347e4904fcf691ee367ef P c850de68a47b4845aa0cca514af1fb5c: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:02:36.559110 2024 tablet_replica.cc:439] T 33c4f0752ba347e4904fcf691ee367ef P 22063e40d22d4fceb578e8f126d3bcd4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c850de68a47b4845aa0cca514af1fb5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } } }
I20260924 03:02:36.559206 2024 tablet_replica.cc:442] T 33c4f0752ba347e4904fcf691ee367ef P 22063e40d22d4fceb578e8f126d3bcd4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:02:36.559892 2028 mvcc.cc:205] Tried to move back new op lower bound from 7332736846050115584 to 7332736846001201152. Current Snapshot: MvccSnapshot[applied={T|T < 7332736846050115584}]
I20260924 03:02:36.560019 2019 tablet_replica.cc:439] T 33c4f0752ba347e4904fcf691ee367ef P ddf11771925343bf872e404886984b3c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c850de68a47b4845aa0cca514af1fb5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 } } peers { permanent_uuid: "c850de68a47b4845aa0cca514af1fb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 } } }
I20260924 03:02:36.560088 2019 tablet_replica.cc:442] T 33c4f0752ba347e4904fcf691ee367ef P ddf11771925343bf872e404886984b3c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:02:36.563999 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:36.561289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35280 (local address 127.0.98.2:41951)
0924 03:02:36.561335 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:36.561337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:36.561390 (+ 53us) server_negotiation.cc:407] Connection header received
0924 03:02:36.561479 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:36.561480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:36.561491 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:36.561544 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:02:36.561653 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.562159 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.563235 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.563474 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.563546 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:36.563733 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:02:36.563781 (+ 48us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:02:36.563847 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:36.563920 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 03:02:36.563941 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:02:36.570088 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:36.566801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51333 (local address 127.0.98.2:41951)
0924 03:02:36.566916 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:36.566918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:36.566957 (+ 39us) server_negotiation.cc:407] Connection header received
0924 03:02:36.567028 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:36.567028 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:36.567036 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:36.567076 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:02:36.567726 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.568196 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.569324 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.569496 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.569547 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:36.569743 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:02:36.569770 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:02:36.569898 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:36.569973 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 03:02:36.570008 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":23}
I20260924 03:02:36.570175 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:36.566722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41951 (local address 127.0.98.2:51333)
0924 03:02:36.566901 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:36.566912 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:36.566973 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:36.567089 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:36.567089 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:36.567090 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:02:36.567647 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:36.567650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.568274 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.568275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:36.569246 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:36.569248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.569566 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.569567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:36.569631 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:36.569645 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:02:36.569912 (+ 267us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:02:36.569914 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:02:36.569970 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 03:02:36.570030 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":2,"thread_start_us":109,"threads_started":1}
I20260924 03:02:36.572261 1664 raft_consensus.cc:1261] T ff9c399265564f10a3519926e9f58411 P 22063e40d22d4fceb578e8f126d3bcd4 [term 1 FOLLOWER]: Refusing update from remote peer c850de68a47b4845aa0cca514af1fb5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:02:36.572254 1944 raft_consensus.cc:1261] T ff9c399265564f10a3519926e9f58411 P ddf11771925343bf872e404886984b3c [term 1 FOLLOWER]: Refusing update from remote peer c850de68a47b4845aa0cca514af1fb5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:02:36.572461 2062 consensus_queue.cc:1035] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22063e40d22d4fceb578e8f126d3bcd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:02:36.572618 2045 consensus_queue.cc:1035] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddf11771925343bf872e404886984b3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:02:36.584595 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:02:36.584683 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:02:36.585263 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e22f16deebe84552be594e3177fe97b3: Sending DeleteTablet for 3 replicas of tablet ff9c399265564f10a3519926e9f58411
I20260924 03:02:36.585570 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet ff9c399265564f10a3519926e9f58411 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:02:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42754
I20260924 03:02:36.585665 2038 tablet_replica.cc:330] T ff9c399265564f10a3519926e9f58411 P 22063e40d22d4fceb578e8f126d3bcd4: stopping tablet replica
I20260924 03:02:36.585716 2038 raft_consensus.cc:2229] T ff9c399265564f10a3519926e9f58411 P 22063e40d22d4fceb578e8f126d3bcd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:02:36.585750 2038 raft_consensus.cc:2258] T ff9c399265564f10a3519926e9f58411 P 22063e40d22d4fceb578e8f126d3bcd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:02:36.585974 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet ff9c399265564f10a3519926e9f58411 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:02:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41464
I20260924 03:02:36.586063 2039 tablet_replica.cc:330] T ff9c399265564f10a3519926e9f58411 P ddf11771925343bf872e404886984b3c: stopping tablet replica
I20260924 03:02:36.586117 2039 raft_consensus.cc:2229] T ff9c399265564f10a3519926e9f58411 P ddf11771925343bf872e404886984b3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:02:36.586161 2039 raft_consensus.cc:2258] T ff9c399265564f10a3519926e9f58411 P ddf11771925343bf872e404886984b3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:02:36.586261 1783 tablet_service.cc:1514] Processing DeleteTablet for tablet ff9c399265564f10a3519926e9f58411 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:02:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35272
I20260924 03:02:36.586359 2037 tablet_replica.cc:330] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c: stopping tablet replica
I20260924 03:02:36.586406 2037 raft_consensus.cc:2229] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:02:36.586488 2037 raft_consensus.cc:2258] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:02:36.586556 2039 ts_tablet_manager.cc:1905] T ff9c399265564f10a3519926e9f58411 P ddf11771925343bf872e404886984b3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:02:36.586738 2038 ts_tablet_manager.cc:1905] T ff9c399265564f10a3519926e9f58411 P 22063e40d22d4fceb578e8f126d3bcd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:02:36.586771 2037 ts_tablet_manager.cc:1905] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:02:36.588320 2039 ts_tablet_manager.cc:1918] T ff9c399265564f10a3519926e9f58411 P ddf11771925343bf872e404886984b3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:02:36.588320 2037 ts_tablet_manager.cc:1918] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:02:36.588331 2038 ts_tablet_manager.cc:1918] T ff9c399265564f10a3519926e9f58411 P 22063e40d22d4fceb578e8f126d3bcd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:02:36.588384 2039 log.cc:1196] T ff9c399265564f10a3519926e9f58411 P ddf11771925343bf872e404886984b3c: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-2/wal/wals/ff9c399265564f10a3519926e9f58411
I20260924 03:02:36.588384 2037 log.cc:1196] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-1/wal/wals/ff9c399265564f10a3519926e9f58411
I20260924 03:02:36.588387 2038 log.cc:1196] T ff9c399265564f10a3519926e9f58411 P 22063e40d22d4fceb578e8f126d3bcd4: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218948286999-392-0/minicluster-data/ts-0/wal/wals/ff9c399265564f10a3519926e9f58411
I20260924 03:02:36.588615 2037 ts_tablet_manager.cc:1939] T ff9c399265564f10a3519926e9f58411 P c850de68a47b4845aa0cca514af1fb5c: Deleting consensus metadata
I20260924 03:02:36.588615 2039 ts_tablet_manager.cc:1939] T ff9c399265564f10a3519926e9f58411 P ddf11771925343bf872e404886984b3c: Deleting consensus metadata
I20260924 03:02:36.588632 2038 ts_tablet_manager.cc:1939] T ff9c399265564f10a3519926e9f58411 P 22063e40d22d4fceb578e8f126d3bcd4: Deleting consensus metadata
I20260924 03:02:36.588964 1493 catalog_manager.cc:4841] TS c850de68a47b4845aa0cca514af1fb5c (127.0.98.2:41951): tablet ff9c399265564f10a3519926e9f58411 (table test-table [id=856f5b1c0b3a44e3804e3eb6e8829bc2]) successfully deleted
I20260924 03:02:36.589002 1492 catalog_manager.cc:4841] TS 22063e40d22d4fceb578e8f126d3bcd4 (127.0.98.1:46425): tablet ff9c399265564f10a3519926e9f58411 (table test-table [id=856f5b1c0b3a44e3804e3eb6e8829bc2]) successfully deleted
I20260924 03:02:36.589035 1493 catalog_manager.cc:4841] TS ddf11771925343bf872e404886984b3c (127.0.98.3:41231): tablet ff9c399265564f10a3519926e9f58411 (table test-table [id=856f5b1c0b3a44e3804e3eb6e8829bc2]) successfully deleted
Sep 24 03:02:36 dist-test-slave-wgsb krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218955, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 03:02:36.596777 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:36.586476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42776 (local address 127.0.98.1:46425)
0924 03:02:36.586540 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:36.586543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:36.587832 (+ 1289us) server_negotiation.cc:407] Connection header received
0924 03:02:36.587940 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:36.587941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:36.587951 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:36.587981 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:36.589014 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.589515 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.590310 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.590532 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.590606 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:36.593725 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:36.593733 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:36.593734 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:36.593740 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:36.595530 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:36.595973 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:36.595976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:36.595977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:36.596076 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:36.596284 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:36.596285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:36.596285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:36.596432 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:36.596544 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:36.596645 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 03:02:36.596710 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 03:02:36.597028 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42776
I20260924 03:02:36.604882 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:36.598098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53618 (local address 127.0.98.62:46523)
0924 03:02:36.598152 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:36.598154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:36.598164 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:02:36.598236 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:36.598237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:36.598246 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:36.598320 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:36.598823 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.599316 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.600205 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:36.600443 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:36.600543 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:36.602033 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:36.602035 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:36.602035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:36.602044 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:36.603492 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:36.603973 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:36.603975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:36.603976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:36.604043 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:36.604279 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:36.604280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:36.604280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:36.604406 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:36.604545 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:36.604668 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 03:02:36.604749 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260924 03:02:36.605156 1507 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53618
I20260924 03:02:36.606005 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 1579
I20260924 03:02:36.612484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 1719
I20260924 03:02:36.617128 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 1859
I20260924 03:02:36.622164 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 1475
2026-09-24T03:02:36Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2818 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:02:36 dist-test-slave-wgsb krb5kdc[2075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:02:36 dist-test-slave-wgsb krb5kdc[2075](info): set up 2 sockets
Sep 24 03:02:36 dist-test-slave-wgsb krb5kdc[2075](info): commencing operation
krb5kdc: starting...
W20260924 03:02:37.667347 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:02:37 dist-test-slave-wgsb krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:02:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:02:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:02:37.787036 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33383
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:37.867192 2088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:37.867446 2088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:37.867483 2088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:37.868683 2088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:02:37.868758 2088 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 03:02:37.868781 2088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:37.868803 2088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:37.868831 2088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:02:37.868855 2088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:02:37.870344 2088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33383
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:37.870633 2088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:37.870874 2088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:02:37.875851 2088 server_base.cc:1034] running on GCE node
W20260924 03:02:37.875948 2093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:37.876030 2094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:37.875833 2096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:37.876346 2088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:37.876652 2088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:37.877815 2088 hybrid_clock.cc:648] HybridClock initialized: now 1790218957877799 us; error 26 us; skew 500 ppm
Sep 24 03:02:37 dist-test-slave-wgsb krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:37.880329 2088 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:02:37.880833 2088 webserver.cc:460] Webserver started at http://127.0.98.62:36277/ using document root <none> and password file <none>
I20260924 03:02:37.881062 2088 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:37.881129 2088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:37.881278 2088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:37.882371 2088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "a20d6abbc75447a69eb1478b7b10f2f2"
format_stamp: "Formatted at 2026-09-24 03:02:37 on dist-test-slave-wgsb"
server_key: "56637d2d501fbe99d21c6ad34d69b887"
server_key_iv: "ce6cfce84a2f048242cbbeb3b45b5b97"
server_key_version: "encryptionkey@0"
I20260924 03:02:37.882738 2088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "a20d6abbc75447a69eb1478b7b10f2f2"
format_stamp: "Formatted at 2026-09-24 03:02:37 on dist-test-slave-wgsb"
server_key: "56637d2d501fbe99d21c6ad34d69b887"
server_key_iv: "ce6cfce84a2f048242cbbeb3b45b5b97"
server_key_version: "encryptionkey@0"
I20260924 03:02:37.884157 2088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:02:37.884927 2103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:37.885141 2088 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:37.885241 2088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "a20d6abbc75447a69eb1478b7b10f2f2"
format_stamp: "Formatted at 2026-09-24 03:02:37 on dist-test-slave-wgsb"
server_key: "56637d2d501fbe99d21c6ad34d69b887"
server_key_iv: "ce6cfce84a2f048242cbbeb3b45b5b97"
server_key_version: "encryptionkey@0"
I20260924 03:02:37.885319 2088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:37.902935 2088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:37.905144 2088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:37.905314 2088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:37.910255 2088 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33383
I20260924 03:02:37.910313 2165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33383 every 8 connection(s)
I20260924 03:02:37.910692 2088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:02:37.911335 2166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:37.911896 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 2088
I20260924 03:02:37.912041 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:02:37.912209 392 external_mini_cluster.cc:1428] Setting key 7c4957077a3594b3f83640f9674392ad
I20260924 03:02:37.914434 2166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a20d6abbc75447a69eb1478b7b10f2f2: Bootstrap starting.
I20260924 03:02:37.915115 2166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a20d6abbc75447a69eb1478b7b10f2f2: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:37.915416 2166 log.cc:824] T 00000000000000000000000000000000 P a20d6abbc75447a69eb1478b7b10f2f2: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:37.916163 2166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a20d6abbc75447a69eb1478b7b10f2f2: No bootstrap required, opened a new log
I20260924 03:02:37.918431 2166 raft_consensus.cc:348] T 00000000000000000000000000000000 P a20d6abbc75447a69eb1478b7b10f2f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20d6abbc75447a69eb1478b7b10f2f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33383 } }
I20260924 03:02:37.918581 2166 raft_consensus.cc:374] T 00000000000000000000000000000000 P a20d6abbc75447a69eb1478b7b10f2f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:37.918604 2166 raft_consensus.cc:729] T 00000000000000000000000000000000 P a20d6abbc75447a69eb1478b7b10f2f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20d6abbc75447a69eb1478b7b10f2f2, State: Initialized, Role: FOLLOWER
I20260924 03:02:37.918741 2166 consensus_queue.cc:260] T 00000000000000000000000000000000 P a20d6abbc75447a69eb1478b7b10f2f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20d6abbc75447a69eb1478b7b10f2f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33383 } }
I20260924 03:02:37.918846 2166 raft_consensus.cc:388] T 00000000000000000000000000000000 P a20d6abbc75447a69eb1478b7b10f2f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:02:37.918895 2166 raft_consensus.cc:482] T 00000000000000000000000000000000 P a20d6abbc75447a69eb1478b7b10f2f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:02:37.918946 2166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a20d6abbc75447a69eb1478b7b10f2f2 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 03:02:37 dist-test-slave-wgsb krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:37.919507 2166 raft_consensus.cc:504] T 00000000000000000000000000000000 P a20d6abbc75447a69eb1478b7b10f2f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20d6abbc75447a69eb1478b7b10f2f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33383 } }
I20260924 03:02:37.919649 2166 leader_election.cc:302] T 00000000000000000000000000000000 P a20d6abbc75447a69eb1478b7b10f2f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a20d6abbc75447a69eb1478b7b10f2f2; no voters:
I20260924 03:02:37.919844 2166 leader_election.cc:288] T 00000000000000000000000000000000 P a20d6abbc75447a69eb1478b7b10f2f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:02:37.919904 2171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a20d6abbc75447a69eb1478b7b10f2f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:37.920046 2171 raft_consensus.cc:686] T 00000000000000000000000000000000 P a20d6abbc75447a69eb1478b7b10f2f2 [term 1 LEADER]: Becoming Leader. State: Replica: a20d6abbc75447a69eb1478b7b10f2f2, State: Running, Role: LEADER
I20260924 03:02:37.920104 2166 sys_catalog.cc:564] T 00000000000000000000000000000000 P a20d6abbc75447a69eb1478b7b10f2f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:02:37.920150 2171 consensus_queue.cc:237] T 00000000000000000000000000000000 P a20d6abbc75447a69eb1478b7b10f2f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20d6abbc75447a69eb1478b7b10f2f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33383 } }
I20260924 03:02:37.920442 2171 sys_catalog.cc:455] T 00000000000000000000000000000000 P a20d6abbc75447a69eb1478b7b10f2f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a20d6abbc75447a69eb1478b7b10f2f2. Latest consensus state: current_term: 1 leader_uuid: "a20d6abbc75447a69eb1478b7b10f2f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20d6abbc75447a69eb1478b7b10f2f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33383 } } }
I20260924 03:02:37.920534 2171 sys_catalog.cc:458] T 00000000000000000000000000000000 P a20d6abbc75447a69eb1478b7b10f2f2 [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:37.920452 2173 sys_catalog.cc:455] T 00000000000000000000000000000000 P a20d6abbc75447a69eb1478b7b10f2f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a20d6abbc75447a69eb1478b7b10f2f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20d6abbc75447a69eb1478b7b10f2f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33383 } } }
I20260924 03:02:37.920627 2173 sys_catalog.cc:458] T 00000000000000000000000000000000 P a20d6abbc75447a69eb1478b7b10f2f2 [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:37.921387 2176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:02:37.921839 2176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:02:37.923956 2176 catalog_manager.cc:1347] Generated new cluster ID: fe36b0e85296498a934d09256625f417
I20260924 03:02:37.924007 2176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:02:37.924055 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:37.912737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59854 (local address 127.0.98.62:33383)
0924 03:02:37.912894 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:37.912903 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:37.912948 (+ 45us) server_negotiation.cc:407] Connection header received
0924 03:02:37.913475 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:37.913484 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:37.913544 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:37.913699 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:37.914393 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:37.915804 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:37.916632 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:37.917325 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:37.917456 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:37.919854 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:37.919858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:37.919858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:37.919871 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:37.922194 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:37.922605 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:37.922608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:37.922608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:37.922700 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:37.923052 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:37.923053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:37.923054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:37.923357 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:37.923502 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:37.923658 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 03:02:37.923794 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":55,"threads_started":1}
I20260924 03:02:37.930408 2176 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:02:37.931133 2176 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:02:37.937330 2176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a20d6abbc75447a69eb1478b7b10f2f2: Generated new TSK 0
I20260924 03:02:37.937515 2176 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:02:37.972286 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33383
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:38.070145 2192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:38.070382 2192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:38.070422 2192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:38.071560 2192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:38.071647 2192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:38.071692 2192 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 03:02:38.071731 2192 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:02:38.071765 2192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:02:38.073391 2192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:38.073675 2192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:38.073908 2192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:38.079879 2198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:38.079902 2192 server_base.cc:1034] running on GCE node
W20260924 03:02:38.079879 2197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:38.079886 2200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:38.080307 2192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:38.080536 2192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:38.081709 2192 hybrid_clock.cc:648] HybridClock initialized: now 1790218958081688 us; error 31 us; skew 500 ppm
Sep 24 03:02:38 dist-test-slave-wgsb krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:38.084535 2192 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:02:38.085028 2192 webserver.cc:460] Webserver started at http://127.0.98.1:43879/ using document root <none> and password file <none>
I20260924 03:02:38.085275 2192 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:38.085350 2192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:38.085522 2192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:38.086690 2192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-0/data/instance:
uuid: "5e7456a33b6a46d5bf984827a286a2f7"
format_stamp: "Formatted at 2026-09-24 03:02:38 on dist-test-slave-wgsb"
server_key: "9aab181dc2a2a3c0411397e6491177ab"
server_key_iv: "24e6aa67480d3c3a09a59088a90a0400"
server_key_version: "encryptionkey@0"
I20260924 03:02:38.087092 2192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5e7456a33b6a46d5bf984827a286a2f7"
format_stamp: "Formatted at 2026-09-24 03:02:38 on dist-test-slave-wgsb"
server_key: "9aab181dc2a2a3c0411397e6491177ab"
server_key_iv: "24e6aa67480d3c3a09a59088a90a0400"
server_key_version: "encryptionkey@0"
I20260924 03:02:38.088673 2192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:02:38.089504 2207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:38.089867 2192 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:02:38.089972 2192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "5e7456a33b6a46d5bf984827a286a2f7"
format_stamp: "Formatted at 2026-09-24 03:02:38 on dist-test-slave-wgsb"
server_key: "9aab181dc2a2a3c0411397e6491177ab"
server_key_iv: "24e6aa67480d3c3a09a59088a90a0400"
server_key_version: "encryptionkey@0"
I20260924 03:02:38.090076 2192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:38.100872 2192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:38.102941 2192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:38.103147 2192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:38.103749 2192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:38.103813 2192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:38.103848 2192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:38.103876 2192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:02:38 dist-test-slave-wgsb krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:38.111069 2192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36911
I20260924 03:02:38.111176 2322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36911 every 8 connection(s)
I20260924 03:02:38.111497 2192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:02:38.114323 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:38.104237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54557 (local address 127.0.98.62:33383)
0924 03:02:38.104285 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:38.104287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:38.104538 (+ 251us) server_negotiation.cc:407] Connection header received
0924 03:02:38.105147 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:38.105148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:38.105164 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:38.105235 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:38.106096 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.106929 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.108668 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.109038 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.109123 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:38.111529 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:38.111531 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:38.111532 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:38.111540 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:38.113006 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:38.113440 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:38.113441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:38.113442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:38.113527 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:38.113770 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:38.113771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:38.113771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:38.113869 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:38.113917 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:38.114187 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 03:02:38.114260 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 03:02:38.114629 2216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:38.104222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.1:54557)
0924 03:02:38.104408 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:38.104442 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:38.105020 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:38.105306 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:38.105310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:38.105719 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:38.105997 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:38.106004 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.107030 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.107031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:38.108543 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:38.108547 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.109134 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.109136 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:38.109244 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:38.109788 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:38.109797 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:38.111420 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:38.113091 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:38.113098 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:38.113100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:38.113358 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:38.113591 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:38.113592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:38.113592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:38.113706 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:38.113928 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:38.113933 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:38.114057 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 03:02:38.114154 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 03:02:38.114293 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":96,"threads_started":1}
I20260924 03:02:38.115360 2323 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33383
I20260924 03:02:38.115480 2323 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:38.115712 2323 heartbeater.cc:511] Master 127.0.98.62:33383 requested a full tablet report, sending...
I20260924 03:02:38.116339 2119 ts_manager.cc:194] Registered new tserver with Master: 5e7456a33b6a46d5bf984827a286a2f7 (127.0.98.1:36911)
I20260924 03:02:38.117195 2119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54557
I20260924 03:02:38.118389 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 2192
I20260924 03:02:38.118468 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:02:38.118652 392 external_mini_cluster.cc:1428] Setting key b0813237e88889ea6b39bdcc633b5d81
I20260924 03:02:38.123262 2329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:38.116580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.1:38191)
0924 03:02:38.116715 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:38.116735 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:38.116789 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:38.116921 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:38.116921 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:38.117087 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:38.117139 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:38.117140 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.118052 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.118053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:38.118829 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:38.118833 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.119280 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.119281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:38.119428 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:38.120018 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:38.120024 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:38.120418 (+ 394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:38.122208 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:38.122209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:38.122209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:38.122424 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:38.122651 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:38.122652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:38.122652 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:38.122707 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:38.122985 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:38.122985 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:38.123027 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 03:02:38.123094 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 03:02:38.123181 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260924 03:02:38.123356 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:38.116643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38191 (local address 127.0.98.62:33383)
0924 03:02:38.116680 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:38.116681 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:38.116794 (+ 113us) server_negotiation.cc:407] Connection header received
0924 03:02:38.116835 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:38.116835 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:38.116844 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:38.116909 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:38.117213 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.117847 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.118966 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.119195 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.119257 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:38.120532 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:38.120533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:38.120533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:38.120541 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:38.122124 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:38.122518 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:38.122519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:38.122519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:38.122580 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:38.122769 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:38.122770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:38.122770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:38.122910 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:38.122974 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:38.123136 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 03:02:38.123221 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:02:38.146608 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33383
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:38.231320 2332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:38.231541 2332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:38.231568 2332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:38.232863 2332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:38.232916 2332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:38.232939 2332 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 03:02:38.232952 2332 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:02:38.232971 2332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:02:38.234472 2332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:38.234681 2332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:38.234877 2332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:38.240355 2340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:38.240355 2337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:38.240348 2338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:38.240665 2332 server_base.cc:1034] running on GCE node
I20260924 03:02:38.240886 2332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:38.241278 2332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:38.242470 2332 hybrid_clock.cc:648] HybridClock initialized: now 1790218958242426 us; error 60 us; skew 500 ppm
Sep 24 03:02:38 dist-test-slave-wgsb krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:38.245640 2332 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:02:38.246199 2332 webserver.cc:460] Webserver started at http://127.0.98.2:45135/ using document root <none> and password file <none>
I20260924 03:02:38.246485 2332 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:38.246568 2332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:38.246740 2332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:38.248078 2332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-1/data/instance:
uuid: "6b4bf12bbdea4f47884ccb78a5476f45"
format_stamp: "Formatted at 2026-09-24 03:02:38 on dist-test-slave-wgsb"
server_key: "4a9f0d77a86740f57744afc9f620ed77"
server_key_iv: "cad1bdb4854a32d602f86e99be17dddc"
server_key_version: "encryptionkey@0"
I20260924 03:02:38.248463 2332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6b4bf12bbdea4f47884ccb78a5476f45"
format_stamp: "Formatted at 2026-09-24 03:02:38 on dist-test-slave-wgsb"
server_key: "4a9f0d77a86740f57744afc9f620ed77"
server_key_iv: "cad1bdb4854a32d602f86e99be17dddc"
server_key_version: "encryptionkey@0"
I20260924 03:02:38.249890 2332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:02:38.250790 2347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:38.251103 2332 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:38.251207 2332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "6b4bf12bbdea4f47884ccb78a5476f45"
format_stamp: "Formatted at 2026-09-24 03:02:38 on dist-test-slave-wgsb"
server_key: "4a9f0d77a86740f57744afc9f620ed77"
server_key_iv: "cad1bdb4854a32d602f86e99be17dddc"
server_key_version: "encryptionkey@0"
I20260924 03:02:38.251297 2332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:38.267486 2332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:38.269658 2332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:38.269840 2332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:38.270480 2332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:38.270545 2332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:38.270581 2332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:38.270601 2332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:02:38 dist-test-slave-wgsb krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:38.278582 2332 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35617
I20260924 03:02:38.278635 2462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35617 every 8 connection(s)
I20260924 03:02:38.279048 2332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:02:38.281317 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:38.270997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53501 (local address 127.0.98.62:33383)
0924 03:02:38.271054 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:38.271057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:38.271217 (+ 160us) server_negotiation.cc:407] Connection header received
0924 03:02:38.271864 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:38.271865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:38.271878 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:38.271940 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:38.272738 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.273304 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.274534 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.274846 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.274954 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:38.278314 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:38.278316 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:38.278316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:38.278324 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:38.280059 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:38.280491 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:38.280491 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:38.280492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:38.280541 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:38.280755 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:38.280756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:38.280756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:38.280858 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:38.280907 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:38.281191 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 03:02:38.281256 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 03:02:38.281584 2356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:38.270853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.2:53501)
0924 03:02:38.271072 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:38.271123 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:38.271750 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:38.272031 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:38.272035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:38.272285 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:38.272634 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:38.272640 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.273414 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.273415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:38.274424 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:38.274426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.274964 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.274965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:38.275114 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:38.275748 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:38.275759 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:38.278186 (+ 2427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:38.280157 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:38.280163 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:38.280165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:38.280395 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:38.280602 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:38.280602 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:38.280603 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:38.280702 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:38.280923 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:38.280927 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:38.281047 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 03:02:38.281136 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 03:02:38.281266 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":118,"threads_started":1}
I20260924 03:02:38.281675 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 2332
I20260924 03:02:38.281752 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:02:38.281967 392 external_mini_cluster.cc:1428] Setting key 60b5275d824d6adf5d6e85e3dc0ac75d
I20260924 03:02:38.282135 2463 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33383
I20260924 03:02:38.282226 2463 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:38.282408 2463 heartbeater.cc:511] Master 127.0.98.62:33383 requested a full tablet report, sending...
I20260924 03:02:38.282756 2119 ts_manager.cc:194] Registered new tserver with Master: 6b4bf12bbdea4f47884ccb78a5476f45 (127.0.98.2:35617)
I20260924 03:02:38.283288 2119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53501
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 03:02:38.290328 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:38.284229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34309 (local address 127.0.98.62:33383)
0924 03:02:38.284308 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:38.284310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:38.284318 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:02:38.284336 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:38.284337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:38.284345 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:38.284421 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:38.284793 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.285330 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.286195 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.286421 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.286510 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:38.287530 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:38.287531 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:38.287532 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:38.287539 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:38.289075 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:38.289445 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:38.289446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:38.289447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:38.289506 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:38.289710 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:38.289711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:38.289711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:38.289912 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:38.289993 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:38.290159 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 03:02:38.290236 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 03:02:38.290341 2469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:38.283223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.2:34309)
0924 03:02:38.283361 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:38.283373 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:38.283433 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:38.284464 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:38.284465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:38.284654 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:38.284707 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:38.284709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.285437 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.285438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:38.286093 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:38.286095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.286511 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.286512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:38.286601 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:38.287087 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:38.287092 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:38.287412 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:38.289165 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:38.289166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:38.289167 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:38.289344 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:38.289579 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:38.289580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:38.289580 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:38.289645 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:38.289996 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:38.289997 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:38.290042 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 03:02:38.290123 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 03:02:38.290232 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:02:38.311342 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33383
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:38.393708 2472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:38.393941 2472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:38.393981 2472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:38.395205 2472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:38.395272 2472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:38.395306 2472 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 03:02:38.395337 2472 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:02:38.395371 2472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:02:38.396911 2472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:38.397193 2472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:38.397432 2472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:02:38.402864 2472 server_base.cc:1034] running on GCE node
W20260924 03:02:38.402817 2480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:38.402827 2478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:38.402825 2477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:38.403318 2472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:38.403544 2472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:38.404752 2472 hybrid_clock.cc:648] HybridClock initialized: now 1790218958404728 us; error 40 us; skew 500 ppm
Sep 24 03:02:38 dist-test-slave-wgsb krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:38.407646 2472 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:02:38.408221 2472 webserver.cc:460] Webserver started at http://127.0.98.3:37789/ using document root <none> and password file <none>
I20260924 03:02:38.408488 2472 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:38.408562 2472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:38.408725 2472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:38.409844 2472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-2/data/instance:
uuid: "d97f3824f995462e83f56ff3c27bf690"
format_stamp: "Formatted at 2026-09-24 03:02:38 on dist-test-slave-wgsb"
server_key: "a58cf3656582346407dee8c95ec165ba"
server_key_iv: "06e5496beabfdc0029e773cfaba92c62"
server_key_version: "encryptionkey@0"
I20260924 03:02:38.410216 2472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d97f3824f995462e83f56ff3c27bf690"
format_stamp: "Formatted at 2026-09-24 03:02:38 on dist-test-slave-wgsb"
server_key: "a58cf3656582346407dee8c95ec165ba"
server_key_iv: "06e5496beabfdc0029e773cfaba92c62"
server_key_version: "encryptionkey@0"
I20260924 03:02:38.411657 2472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:02:38.412461 2487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:38.412662 2472 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:02:38.412763 2472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "d97f3824f995462e83f56ff3c27bf690"
format_stamp: "Formatted at 2026-09-24 03:02:38 on dist-test-slave-wgsb"
server_key: "a58cf3656582346407dee8c95ec165ba"
server_key_iv: "06e5496beabfdc0029e773cfaba92c62"
server_key_version: "encryptionkey@0"
I20260924 03:02:38.412851 2472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:38.424649 2472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:38.427163 2472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:38.427335 2472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:38.428018 2472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:38.428081 2472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:38.428119 2472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:38.428143 2472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:02:38 dist-test-slave-wgsb krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:38.436205 2472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39243
I20260924 03:02:38.436288 2602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39243 every 8 connection(s)
I20260924 03:02:38.436707 2472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:02:38.438822 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:38.428462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51341 (local address 127.0.98.62:33383)
0924 03:02:38.428534 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:38.428537 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:38.428799 (+ 262us) server_negotiation.cc:407] Connection header received
0924 03:02:38.429334 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:38.429336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:38.429351 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:38.429433 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:38.430154 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.431203 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.432365 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.432754 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.432842 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:38.435577 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:38.435579 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:38.435580 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:38.435588 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:38.437443 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:38.437833 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:38.437834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:38.437834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:38.437890 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:38.438163 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:38.438164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:38.438164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:38.438280 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:38.438351 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:38.438655 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 03:02:38.438732 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 03:02:38.439023 2496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:38.428458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.3:51341)
0924 03:02:38.428645 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:38.428687 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:38.429220 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:38.429508 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:38.429511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:38.429849 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:38.430070 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:38.430075 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.431307 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.431308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:38.432249 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:38.432252 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.432870 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.432871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:38.433031 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:38.433601 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:38.433612 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:38.435472 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:38.437523 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:38.437528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:38.437530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:38.437751 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:38.437966 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:38.437967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:38.437968 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:38.438086 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:38.438356 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:38.438362 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:38.438502 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 03:02:38.438621 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 03:02:38.438782 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":123,"threads_started":1}
I20260924 03:02:38.439540 2603 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33383
I20260924 03:02:38.439641 2603 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:38.439821 2603 heartbeater.cc:511] Master 127.0.98.62:33383 requested a full tablet report, sending...
I20260924 03:02:38.440299 2119 ts_manager.cc:194] Registered new tserver with Master: d97f3824f995462e83f56ff3c27bf690 (127.0.98.3:39243)
I20260924 03:02:38.440836 2119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51341
I20260924 03:02:38.446151 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 2472
I20260924 03:02:38.446221 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:02:38.446446 392 external_mini_cluster.cc:1428] Setting key 8fa6d94f4fa81e4e2df4c2e374eb4f90
I20260924 03:02:38.447319 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:38.440989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57353 (local address 127.0.98.62:33383)
0924 03:02:38.441033 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:38.441035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:38.441048 (+ 13us) server_negotiation.cc:407] Connection header received
0924 03:02:38.441129 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:38.441130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:38.441139 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:38.441182 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:38.441582 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.442113 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.442938 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.443135 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.443200 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:38.444349 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:38.444350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:38.444350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:38.444357 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:38.445924 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:38.446403 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:38.446404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:38.446405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:38.446475 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:38.446693 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:38.446694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:38.446694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:38.446842 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:38.446908 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:38.447152 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 03:02:38.447224 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:02:38.447343 2609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:38.440784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.3:57353)
0924 03:02:38.440996 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:38.441009 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:38.441058 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:38.441187 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:38.441187 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:38.441428 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:38.441495 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:38.441496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.442215 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.442216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:38.442809 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:38.442811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.443199 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.443200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:38.443266 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:38.443776 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:38.443780 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:38.444273 (+ 493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:38.446020 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:38.446021 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:38.446021 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:38.446325 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:38.446536 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:38.446537 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:38.446537 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:38.446611 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:38.446965 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:38.446966 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:38.447028 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 03:02:38.447112 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 03:02:38.447199 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":120,"threads_started":1}
I20260924 03:02:38.447364 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 03:02:38 dist-test-slave-wgsb krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 03:02:38 dist-test-slave-wgsb krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:38.470960 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:38.463801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59866 (local address 127.0.98.62:33383)
0924 03:02:38.463865 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:38.463867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:38.463877 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:02:38.463911 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:38.463912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:38.463922 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:38.463973 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:38.464550 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.465073 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.465817 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.466009 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.466071 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:38.468566 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:38.468568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:38.468568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:38.468575 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:38.469930 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:38.470264 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:38.470265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:38.470265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:38.470318 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:38.470499 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:38.470500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:38.470500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:38.470610 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:38.470687 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:38.470834 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 03:02:38.470894 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 03:02:38.473304 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:02:38.474992 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:02:38.481731 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:38.477393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35617 (local address 127.0.0.1:43498)
0924 03:02:38.477594 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:38.477636 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:38.477702 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:38.478211 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:38.478212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:38.478214 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:38.478267 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:38.478269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.479636 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.479637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:38.480616 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:38.480617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.481489 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.481490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:38.481558 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:38.481562 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:02:38.481614 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 03:02:38.481658 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":74,"threads_started":1}
I20260924 03:02:38.481844 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:38.477625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45510 (local address 127.0.98.1:36911)
0924 03:02:38.477933 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:38.477937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:38.477944 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:02:38.477975 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:38.477977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:38.477989 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:38.478094 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:38.478319 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.479010 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.480462 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.481193 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.481273 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:38.481331 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:38.481703 (+ 372us) server_negotiation.cc:299] Negotiation successful
0924 03:02:38.481748 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":85,"threads_started":1}
I20260924 03:02:38.481647 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:38.477453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36911 (local address 127.0.0.1:45510)
0924 03:02:38.477668 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:38.477680 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:38.477736 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:38.478109 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:38.478112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:38.478114 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:38.478187 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:38.478188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.479155 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.479156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:38.480383 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:38.480386 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.481283 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.481284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:38.481361 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:38.481375 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 03:02:38.481476 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 03:02:38.481541 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":84,"threads_started":1}
I20260924 03:02:38.482167 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:38.477490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43498 (local address 127.0.98.2:35617)
0924 03:02:38.478035 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:38.478039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:38.478048 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:02:38.478084 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:38.478085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:38.478099 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:38.478342 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:38.478400 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.479517 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.480695 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.481386 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.481476 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:38.481542 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:38.481644 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 03:02:38.481688 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":95,"threads_started":1}
I20260924 03:02:38.482784 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:38.477622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39243 (local address 127.0.0.1:39402)
0924 03:02:38.477793 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:38.477802 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:38.477846 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:38.478264 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:38.478265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:38.478266 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:38.478304 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:38.478305 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.480683 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.480684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:38.481404 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:38.481407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.482533 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.482534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:38.482596 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:38.482601 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:02:38.482668 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 03:02:38.482707 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":76,"threads_started":1}
I20260924 03:02:38.482654 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 6dc7692ff4b740528e95f6f231e550bc (DEFAULT_TABLE table=test-table [id=f52e653e194f418a8f1d25896ef1736b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:38.482811 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:38.477813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39402 (local address 127.0.98.3:39243)
0924 03:02:38.477976 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:38.477981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:38.477993 (+ 12us) server_negotiation.cc:407] Connection header received
0924 03:02:38.478043 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:38.478045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:38.478128 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:38.478200 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:38.478374 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.479517 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.481491 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.482448 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.482521 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:38.482573 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:38.482687 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 03:02:38.482724 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260924 03:02:38.482788 2397 tablet_service.cc:1467] Processing CreateTablet for tablet 6dc7692ff4b740528e95f6f231e550bc (DEFAULT_TABLE table=test-table [id=f52e653e194f418a8f1d25896ef1736b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:38.483031 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dc7692ff4b740528e95f6f231e550bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:38.483114 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dc7692ff4b740528e95f6f231e550bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:38.483814 2537 tablet_service.cc:1467] Processing CreateTablet for tablet 6dc7692ff4b740528e95f6f231e550bc (DEFAULT_TABLE table=test-table [id=f52e653e194f418a8f1d25896ef1736b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:38.484139 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dc7692ff4b740528e95f6f231e550bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:38.486227 2626 tablet_bootstrap.cc:492] T 6dc7692ff4b740528e95f6f231e550bc P 5e7456a33b6a46d5bf984827a286a2f7: Bootstrap starting.
I20260924 03:02:38.486982 2627 tablet_bootstrap.cc:492] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45: Bootstrap starting.
I20260924 03:02:38.487178 2626 tablet_bootstrap.cc:654] T 6dc7692ff4b740528e95f6f231e550bc P 5e7456a33b6a46d5bf984827a286a2f7: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:38.487417 2628 tablet_bootstrap.cc:492] T 6dc7692ff4b740528e95f6f231e550bc P d97f3824f995462e83f56ff3c27bf690: Bootstrap starting.
I20260924 03:02:38.487679 2627 tablet_bootstrap.cc:654] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:38.487735 2626 log.cc:824] T 6dc7692ff4b740528e95f6f231e550bc P 5e7456a33b6a46d5bf984827a286a2f7: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:38.488029 2627 log.cc:824] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:38.488444 2628 tablet_bootstrap.cc:654] T 6dc7692ff4b740528e95f6f231e550bc P d97f3824f995462e83f56ff3c27bf690: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:38.488642 2626 tablet_bootstrap.cc:492] T 6dc7692ff4b740528e95f6f231e550bc P 5e7456a33b6a46d5bf984827a286a2f7: No bootstrap required, opened a new log
I20260924 03:02:38.488710 2627 tablet_bootstrap.cc:492] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45: No bootstrap required, opened a new log
I20260924 03:02:38.488718 2626 ts_tablet_manager.cc:1397] T 6dc7692ff4b740528e95f6f231e550bc P 5e7456a33b6a46d5bf984827a286a2f7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 03:02:38.488770 2627 ts_tablet_manager.cc:1397] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 03:02:38.488809 2628 log.cc:824] T 6dc7692ff4b740528e95f6f231e550bc P d97f3824f995462e83f56ff3c27bf690: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:38.489485 2628 tablet_bootstrap.cc:492] T 6dc7692ff4b740528e95f6f231e550bc P d97f3824f995462e83f56ff3c27bf690: No bootstrap required, opened a new log
I20260924 03:02:38.489914 2628 ts_tablet_manager.cc:1397] T 6dc7692ff4b740528e95f6f231e550bc P d97f3824f995462e83f56ff3c27bf690: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 03:02:38.491401 2628 raft_consensus.cc:348] T 6dc7692ff4b740528e95f6f231e550bc P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } } peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } }
I20260924 03:02:38.491575 2628 raft_consensus.cc:374] T 6dc7692ff4b740528e95f6f231e550bc P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:38.491657 2628 raft_consensus.cc:729] T 6dc7692ff4b740528e95f6f231e550bc P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d97f3824f995462e83f56ff3c27bf690, State: Initialized, Role: FOLLOWER
I20260924 03:02:38.491798 2628 consensus_queue.cc:260] T 6dc7692ff4b740528e95f6f231e550bc P d97f3824f995462e83f56ff3c27bf690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } } peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } }
I20260924 03:02:38.491854 2626 raft_consensus.cc:348] T 6dc7692ff4b740528e95f6f231e550bc P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } } peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } }
I20260924 03:02:38.491849 2627 raft_consensus.cc:348] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } } peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } }
I20260924 03:02:38.492010 2626 raft_consensus.cc:374] T 6dc7692ff4b740528e95f6f231e550bc P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:38.492010 2627 raft_consensus.cc:374] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:38.492063 2627 raft_consensus.cc:729] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b4bf12bbdea4f47884ccb78a5476f45, State: Initialized, Role: FOLLOWER
I20260924 03:02:38.492064 2626 raft_consensus.cc:729] T 6dc7692ff4b740528e95f6f231e550bc P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7456a33b6a46d5bf984827a286a2f7, State: Initialized, Role: FOLLOWER
I20260924 03:02:38.492111 2628 ts_tablet_manager.cc:1428] T 6dc7692ff4b740528e95f6f231e550bc P d97f3824f995462e83f56ff3c27bf690: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 03:02:38.492285 2603 heartbeater.cc:503] Master 127.0.98.62:33383 was elected leader, sending a full tablet report...
I20260924 03:02:38.492185 2627 consensus_queue.cc:260] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } } peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } }
I20260924 03:02:38.492210 2626 consensus_queue.cc:260] T 6dc7692ff4b740528e95f6f231e550bc P 5e7456a33b6a46d5bf984827a286a2f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } } peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } }
I20260924 03:02:38.492740 2627 ts_tablet_manager.cc:1428] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260924 03:02:38.492820 2323 heartbeater.cc:503] Master 127.0.98.62:33383 was elected leader, sending a full tablet report...
I20260924 03:02:38.492871 2463 heartbeater.cc:503] Master 127.0.98.62:33383 was elected leader, sending a full tablet report...
I20260924 03:02:38.492780 2626 ts_tablet_manager.cc:1428] T 6dc7692ff4b740528e95f6f231e550bc P 5e7456a33b6a46d5bf984827a286a2f7: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260924 03:02:38.529680 2464 tablet.cc:2369] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:02:38.612562 2324 tablet.cc:2369] T 6dc7692ff4b740528e95f6f231e550bc P 5e7456a33b6a46d5bf984827a286a2f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:02:38.687465 2604 tablet.cc:2369] T 6dc7692ff4b740528e95f6f231e550bc P d97f3824f995462e83f56ff3c27bf690: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:02:38.815038 2633 raft_consensus.cc:482] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:02:38.815165 2633 raft_consensus.cc:504] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } } peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } }
I20260924 03:02:38.815565 2633 leader_election.cc:288] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d97f3824f995462e83f56ff3c27bf690 (127.0.98.3:39243), 5e7456a33b6a46d5bf984827a286a2f7 (127.0.98.1:36911)
I20260924 03:02:38.818747 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:38.815890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37173 (local address 127.0.98.1:36911)
0924 03:02:38.815944 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:38.815946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:38.816147 (+ 201us) server_negotiation.cc:407] Connection header received
0924 03:02:38.816266 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:38.816268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:38.816281 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:38.816333 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:38.816550 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.817021 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.818028 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.818395 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.818482 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:38.818494 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:38.818636 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 03:02:38.818670 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 03:02:38.818778 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:38.815904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35565 (local address 127.0.98.3:39243)
0924 03:02:38.815953 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:38.815955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:38.816177 (+ 222us) server_negotiation.cc:407] Connection header received
0924 03:02:38.816252 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:38.816252 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:38.816265 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:38.816359 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:38.816524 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.816989 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.818046 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.818433 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.818515 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:38.818524 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:38.818675 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 03:02:38.818710 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 03:02:38.818799 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:38.815817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36911 (local address 127.0.98.2:37173)
0924 03:02:38.816057 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:38.816077 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:38.816168 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:38.816336 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:38.816337 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:38.816339 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:38.816408 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:38.816410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.817117 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.817118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:38.817936 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:38.817939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.818482 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.818483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:38.818559 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:38.818562 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:02:38.818630 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 03:02:38.818670 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":22,"thread_start_us":70,"threads_started":1}
I20260924 03:02:38.818811 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:38.815817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39243 (local address 127.0.98.2:35565)
0924 03:02:38.816082 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:38.816094 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:38.816194 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:38.816365 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:38.816366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:38.816368 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:38.816431 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:38.816433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.817118 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.817119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:38.817937 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:38.817939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.818516 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.818517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:38.818585 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:38.818589 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:02:38.818666 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 03:02:38.818705 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":41,"thread_start_us":67,"threads_started":1}
I20260924 03:02:38.819165 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc7692ff4b740528e95f6f231e550bc" candidate_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e7456a33b6a46d5bf984827a286a2f7" is_pre_election: true
I20260924 03:02:38.819314 2277 raft_consensus.cc:2454] T 6dc7692ff4b740528e95f6f231e550bc P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b4bf12bbdea4f47884ccb78a5476f45 in term 0.
I20260924 03:02:38.819288 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc7692ff4b740528e95f6f231e550bc" candidate_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d97f3824f995462e83f56ff3c27bf690" is_pre_election: true
I20260924 03:02:38.819428 2557 raft_consensus.cc:2454] T 6dc7692ff4b740528e95f6f231e550bc P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b4bf12bbdea4f47884ccb78a5476f45 in term 0.
I20260924 03:02:38.819684 2350 leader_election.cc:302] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b4bf12bbdea4f47884ccb78a5476f45, d97f3824f995462e83f56ff3c27bf690; no voters:
I20260924 03:02:38.819907 2633 raft_consensus.cc:2781] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:02:38.820029 2633 raft_consensus.cc:482] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:02:38.820050 2633 raft_consensus.cc:3037] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:38.820797 2633 raft_consensus.cc:504] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } } peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } }
I20260924 03:02:38.821012 2633 leader_election.cc:288] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45 [CANDIDATE]: Term 1 election: Requested vote from peers d97f3824f995462e83f56ff3c27bf690 (127.0.98.3:39243), 5e7456a33b6a46d5bf984827a286a2f7 (127.0.98.1:36911)
I20260924 03:02:38.821100 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc7692ff4b740528e95f6f231e550bc" candidate_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e7456a33b6a46d5bf984827a286a2f7"
I20260924 03:02:38.821170 2277 raft_consensus.cc:3037] T 6dc7692ff4b740528e95f6f231e550bc P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:38.821166 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc7692ff4b740528e95f6f231e550bc" candidate_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d97f3824f995462e83f56ff3c27bf690"
I20260924 03:02:38.821240 2557 raft_consensus.cc:3037] T 6dc7692ff4b740528e95f6f231e550bc P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:38.821880 2277 raft_consensus.cc:2454] T 6dc7692ff4b740528e95f6f231e550bc P 5e7456a33b6a46d5bf984827a286a2f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b4bf12bbdea4f47884ccb78a5476f45 in term 1.
I20260924 03:02:38.822005 2557 raft_consensus.cc:2454] T 6dc7692ff4b740528e95f6f231e550bc P d97f3824f995462e83f56ff3c27bf690 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b4bf12bbdea4f47884ccb78a5476f45 in term 1.
I20260924 03:02:38.822080 2349 leader_election.cc:302] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e7456a33b6a46d5bf984827a286a2f7, 6b4bf12bbdea4f47884ccb78a5476f45; no voters:
I20260924 03:02:38.822182 2633 raft_consensus.cc:2781] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:38.822312 2633 raft_consensus.cc:686] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45 [term 1 LEADER]: Becoming Leader. State: Replica: 6b4bf12bbdea4f47884ccb78a5476f45, State: Running, Role: LEADER
I20260924 03:02:38.822422 2633 consensus_queue.cc:237] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } } peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } }
I20260924 03:02:38.823230 2119 catalog_manager.cc:5495] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b4bf12bbdea4f47884ccb78a5476f45 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } health_report { overall_health: UNKNOWN } } }
Sep 24 03:02:38 dist-test-slave-wgsb krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:38.870064 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:38.856359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.0.1:59878)
0924 03:02:38.856553 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:38.856564 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:38.856633 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:38.856774 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:38.856775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:38.857077 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:38.857128 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:38.857130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.857770 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.857771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:38.858415 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:38.858418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.858812 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.858813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:38.858862 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:38.865430 (+ 6568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:38.865435 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:38.867030 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:38.868888 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:38.868896 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:38.868899 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:38.869139 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:38.869349 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:38.869349 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:38.869350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:38.869440 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:38.869723 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:38.869729 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:38.869780 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 03:02:38.869846 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 03:02:38.869959 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":66,"threads_started":1}
I20260924 03:02:38.870146 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:38.856497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59878 (local address 127.0.98.62:33383)
0924 03:02:38.856575 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:38.856577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:38.856627 (+ 50us) server_negotiation.cc:407] Connection header received
0924 03:02:38.856693 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:38.856694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:38.856702 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:38.856745 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:38.857198 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.857693 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.858526 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:38.858709 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:38.858779 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:38.867149 (+ 8370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:38.867150 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:38.867151 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:38.867158 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:38.868774 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:38.869223 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:38.869224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:38.869225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:38.869280 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:38.869504 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:38.869504 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:38.869504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:38.869643 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:38.869699 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:38.869878 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 03:02:38.870008 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":57}
I20260924 03:02:38.872527 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59878:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 03:02:38.872725 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:02:38.874401 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 359a5d64e02445eaafa86a640015964a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3ffb601d77f24aada5b21a2ef8248a68]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:02:38.874419 2397 tablet_service.cc:1467] Processing CreateTablet for tablet 359a5d64e02445eaafa86a640015964a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3ffb601d77f24aada5b21a2ef8248a68]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:02:38.874584 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 359a5d64e02445eaafa86a640015964a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:38.874585 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 359a5d64e02445eaafa86a640015964a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:38.876209 2626 tablet_bootstrap.cc:492] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7: Bootstrap starting.
I20260924 03:02:38.876211 2627 tablet_bootstrap.cc:492] T 359a5d64e02445eaafa86a640015964a P 6b4bf12bbdea4f47884ccb78a5476f45: Bootstrap starting.
I20260924 03:02:38.876276 2537 tablet_service.cc:1467] Processing CreateTablet for tablet 359a5d64e02445eaafa86a640015964a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3ffb601d77f24aada5b21a2ef8248a68]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:02:38.876430 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 359a5d64e02445eaafa86a640015964a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:38.876719 2626 tablet_bootstrap.cc:654] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:38.876719 2627 tablet_bootstrap.cc:654] T 359a5d64e02445eaafa86a640015964a P 6b4bf12bbdea4f47884ccb78a5476f45: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:38.877467 2626 tablet_bootstrap.cc:492] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7: No bootstrap required, opened a new log
I20260924 03:02:38.877525 2626 ts_tablet_manager.cc:1397] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:38.877740 2628 tablet_bootstrap.cc:492] T 359a5d64e02445eaafa86a640015964a P d97f3824f995462e83f56ff3c27bf690: Bootstrap starting.
I20260924 03:02:38.877767 2627 tablet_bootstrap.cc:492] T 359a5d64e02445eaafa86a640015964a P 6b4bf12bbdea4f47884ccb78a5476f45: No bootstrap required, opened a new log
I20260924 03:02:38.877712 2626 raft_consensus.cc:348] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:38.877806 2626 raft_consensus.cc:374] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:38.877812 2627 ts_tablet_manager.cc:1397] T 359a5d64e02445eaafa86a640015964a P 6b4bf12bbdea4f47884ccb78a5476f45: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260924 03:02:38.877833 2626 raft_consensus.cc:729] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7456a33b6a46d5bf984827a286a2f7, State: Initialized, Role: FOLLOWER
I20260924 03:02:38.877890 2626 consensus_queue.cc:260] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:38.878007 2626 ts_tablet_manager.cc:1428] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 03:02:38.877982 2627 raft_consensus.cc:348] T 359a5d64e02445eaafa86a640015964a P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:38.878052 2627 raft_consensus.cc:374] T 359a5d64e02445eaafa86a640015964a P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:38.878078 2627 raft_consensus.cc:729] T 359a5d64e02445eaafa86a640015964a P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b4bf12bbdea4f47884ccb78a5476f45, State: Initialized, Role: FOLLOWER
I20260924 03:02:38.878078 2634 tablet_replica.cc:439] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } } }
I20260924 03:02:38.878129 2627 consensus_queue.cc:260] T 359a5d64e02445eaafa86a640015964a P 6b4bf12bbdea4f47884ccb78a5476f45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:38.878176 2634 tablet_replica.cc:442] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:02:38.878221 2627 ts_tablet_manager.cc:1428] T 359a5d64e02445eaafa86a640015964a P 6b4bf12bbdea4f47884ccb78a5476f45: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:38.878296 2628 tablet_bootstrap.cc:654] T 359a5d64e02445eaafa86a640015964a P d97f3824f995462e83f56ff3c27bf690: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:38.878392 2633 tablet_replica.cc:439] T 359a5d64e02445eaafa86a640015964a P 6b4bf12bbdea4f47884ccb78a5476f45: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } } }
I20260924 03:02:38.878479 2633 tablet_replica.cc:442] T 359a5d64e02445eaafa86a640015964a P 6b4bf12bbdea4f47884ccb78a5476f45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:02:38.878998 2628 tablet_bootstrap.cc:492] T 359a5d64e02445eaafa86a640015964a P d97f3824f995462e83f56ff3c27bf690: No bootstrap required, opened a new log
I20260924 03:02:38.879079 2628 ts_tablet_manager.cc:1397] T 359a5d64e02445eaafa86a640015964a P d97f3824f995462e83f56ff3c27bf690: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:38.879232 2628 raft_consensus.cc:348] T 359a5d64e02445eaafa86a640015964a P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:38.879298 2628 raft_consensus.cc:374] T 359a5d64e02445eaafa86a640015964a P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:38.879313 2628 raft_consensus.cc:729] T 359a5d64e02445eaafa86a640015964a P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d97f3824f995462e83f56ff3c27bf690, State: Initialized, Role: FOLLOWER
I20260924 03:02:38.879374 2628 consensus_queue.cc:260] T 359a5d64e02445eaafa86a640015964a P d97f3824f995462e83f56ff3c27bf690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:38.879482 2628 ts_tablet_manager.cc:1428] T 359a5d64e02445eaafa86a640015964a P d97f3824f995462e83f56ff3c27bf690: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:38.879561 2632 tablet_replica.cc:439] T 359a5d64e02445eaafa86a640015964a P d97f3824f995462e83f56ff3c27bf690: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } } }
I20260924 03:02:38.879666 2632 tablet_replica.cc:442] T 359a5d64e02445eaafa86a640015964a P d97f3824f995462e83f56ff3c27bf690: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:02:39.022754 2634 raft_consensus.cc:482] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:02:39.022871 2634 raft_consensus.cc:504] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:39.023269 2634 leader_election.cc:288] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b4bf12bbdea4f47884ccb78a5476f45 (127.0.98.2:35617), d97f3824f995462e83f56ff3c27bf690 (127.0.98.3:39243)
I20260924 03:02:39.026341 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:39.023458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35617 (local address 127.0.98.1:37667)
0924 03:02:39.023627 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:39.023640 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:39.023694 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:39.023937 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:39.023939 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:39.023942 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:39.024021 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:39.024024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.024696 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.024697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:39.025539 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:39.025541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.026127 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.026131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:39.026180 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:39.026183 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:02:39.026249 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 03:02:39.026280 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260924 03:02:39.026400 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:39.023648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37667 (local address 127.0.98.2:35617)
0924 03:02:39.023790 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:39.023793 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:39.023803 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:02:39.023832 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:39.023833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:39.023845 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:39.023926 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:39.024103 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.024611 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.025622 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.025995 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.026085 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:39.026098 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:39.026256 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 03:02:39.026306 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260924 03:02:39.026721 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:39.023801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55259 (local address 127.0.98.3:39243)
0924 03:02:39.023867 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:39.023870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:39.023878 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:02:39.023933 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:39.023934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:39.023943 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:39.024003 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:39.024178 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.024626 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.025660 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.025994 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.026079 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:39.026088 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:39.026621 (+ 533us) server_negotiation.cc:299] Negotiation successful
0924 03:02:39.026660 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 03:02:39.026731 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:39.023707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39243 (local address 127.0.98.1:55259)
0924 03:02:39.023788 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:39.023800 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:39.023862 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:39.024018 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:39.024019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:39.024020 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:39.024106 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:39.024107 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.024749 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.024750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:39.025538 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:39.025540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.026387 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.026389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:39.026519 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:39.026524 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:02:39.026612 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 03:02:39.026647 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":42,"thread_start_us":44,"threads_started":1}
I20260924 03:02:39.026924 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "359a5d64e02445eaafa86a640015964a" candidate_uuid: "5e7456a33b6a46d5bf984827a286a2f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d97f3824f995462e83f56ff3c27bf690" is_pre_election: true
I20260924 03:02:39.026960 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "359a5d64e02445eaafa86a640015964a" candidate_uuid: "5e7456a33b6a46d5bf984827a286a2f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" is_pre_election: true
I20260924 03:02:39.027014 2557 raft_consensus.cc:2454] T 359a5d64e02445eaafa86a640015964a P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e7456a33b6a46d5bf984827a286a2f7 in term 0.
I20260924 03:02:39.027055 2417 raft_consensus.cc:2454] T 359a5d64e02445eaafa86a640015964a P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e7456a33b6a46d5bf984827a286a2f7 in term 0.
I20260924 03:02:39.027243 2210 leader_election.cc:302] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e7456a33b6a46d5bf984827a286a2f7, 6b4bf12bbdea4f47884ccb78a5476f45; no voters:
I20260924 03:02:39.027364 2634 raft_consensus.cc:2781] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:02:39.027405 2634 raft_consensus.cc:482] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:02:39.027429 2634 raft_consensus.cc:3037] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:39.028169 2634 raft_consensus.cc:504] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:39.028295 2634 leader_election.cc:288] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7 [CANDIDATE]: Term 1 election: Requested vote from peers 6b4bf12bbdea4f47884ccb78a5476f45 (127.0.98.2:35617), d97f3824f995462e83f56ff3c27bf690 (127.0.98.3:39243)
I20260924 03:02:39.028434 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "359a5d64e02445eaafa86a640015964a" candidate_uuid: "5e7456a33b6a46d5bf984827a286a2f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b4bf12bbdea4f47884ccb78a5476f45"
I20260924 03:02:39.028445 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "359a5d64e02445eaafa86a640015964a" candidate_uuid: "5e7456a33b6a46d5bf984827a286a2f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d97f3824f995462e83f56ff3c27bf690"
I20260924 03:02:39.028514 2557 raft_consensus.cc:3037] T 359a5d64e02445eaafa86a640015964a P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:39.028537 2417 raft_consensus.cc:3037] T 359a5d64e02445eaafa86a640015964a P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:39.029105 2557 raft_consensus.cc:2454] T 359a5d64e02445eaafa86a640015964a P d97f3824f995462e83f56ff3c27bf690 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e7456a33b6a46d5bf984827a286a2f7 in term 1.
I20260924 03:02:39.029253 2210 leader_election.cc:302] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e7456a33b6a46d5bf984827a286a2f7, d97f3824f995462e83f56ff3c27bf690; no voters:
I20260924 03:02:39.029352 2634 raft_consensus.cc:2781] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:39.029363 2417 raft_consensus.cc:2454] T 359a5d64e02445eaafa86a640015964a P 6b4bf12bbdea4f47884ccb78a5476f45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e7456a33b6a46d5bf984827a286a2f7 in term 1.
I20260924 03:02:39.029457 2634 raft_consensus.cc:686] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7 [term 1 LEADER]: Becoming Leader. State: Replica: 5e7456a33b6a46d5bf984827a286a2f7, State: Running, Role: LEADER
I20260924 03:02:39.029552 2634 consensus_queue.cc:237] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:39.029959 2651 tablet_replica.cc:439] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7: TxnStatusTablet state changed. Reason: New leader 5e7456a33b6a46d5bf984827a286a2f7. Latest consensus state: current_term: 1 leader_uuid: "5e7456a33b6a46d5bf984827a286a2f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } } }
I20260924 03:02:39.030066 2651 tablet_replica.cc:442] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:02:39.030293 2653 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 03:02:39.030398 2653 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 03:02:39.030346 2119 catalog_manager.cc:5495] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e7456a33b6a46d5bf984827a286a2f7 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5e7456a33b6a46d5bf984827a286a2f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } health_report { overall_health: UNKNOWN } } }
I20260924 03:02:39.061488 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:39.058902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36911 (local address 127.0.0.1:45518)
0924 03:02:39.058979 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:39.059001 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:39.059057 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:39.059245 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:39.059246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:39.059247 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:02:39.059299 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:39.059301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.060009 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.060010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:39.060590 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:39.060591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.060945 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.060946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:39.061021 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:39.061042 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:02:39.061312 (+ 270us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:02:39.061320 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 03:02:39.061366 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 03:02:39.061418 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 03:02:39.061466 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:39.058962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45518 (local address 127.0.98.1:36911)
0924 03:02:39.059013 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:39.059015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:39.059039 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:02:39.059133 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:39.059134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:39.059155 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:39.059239 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:02:39.059362 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.059908 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.060673 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.060863 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.060946 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:39.061135 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:02:39.061237 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:02:39.061323 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:39.061367 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 03:02:39.061387 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 03:02:39.063453 2417 raft_consensus.cc:1261] T 359a5d64e02445eaafa86a640015964a P 6b4bf12bbdea4f47884ccb78a5476f45 [term 1 FOLLOWER]: Refusing update from remote peer 5e7456a33b6a46d5bf984827a286a2f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:02:39.063454 2557 raft_consensus.cc:1261] T 359a5d64e02445eaafa86a640015964a P d97f3824f995462e83f56ff3c27bf690 [term 1 FOLLOWER]: Refusing update from remote peer 5e7456a33b6a46d5bf984827a286a2f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:02:39.063844 2634 consensus_queue.cc:1035] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:02:39.064033 2651 consensus_queue.cc:1035] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:02:39.065236 2632 tablet_replica.cc:439] T 359a5d64e02445eaafa86a640015964a P d97f3824f995462e83f56ff3c27bf690: TxnStatusTablet state changed. Reason: New leader 5e7456a33b6a46d5bf984827a286a2f7. Latest consensus state: current_term: 1 leader_uuid: "5e7456a33b6a46d5bf984827a286a2f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } } }
I20260924 03:02:39.065243 2633 tablet_replica.cc:439] T 359a5d64e02445eaafa86a640015964a P 6b4bf12bbdea4f47884ccb78a5476f45: TxnStatusTablet state changed. Reason: New leader 5e7456a33b6a46d5bf984827a286a2f7. Latest consensus state: current_term: 1 leader_uuid: "5e7456a33b6a46d5bf984827a286a2f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } } }
I20260924 03:02:39.065323 2632 tablet_replica.cc:442] T 359a5d64e02445eaafa86a640015964a P d97f3824f995462e83f56ff3c27bf690: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:02:39.065327 2633 tablet_replica.cc:442] T 359a5d64e02445eaafa86a640015964a P 6b4bf12bbdea4f47884ccb78a5476f45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:02:39.066202 2651 tablet_replica.cc:439] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e7456a33b6a46d5bf984827a286a2f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } } }
I20260924 03:02:39.066280 2651 tablet_replica.cc:442] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:02:39.066393 2656 mvcc.cc:205] Tried to move back new op lower bound from 7332736856320823296 to 7332736856185384960. Current Snapshot: MvccSnapshot[applied={T|T < 7332736856320823296}]
I20260924 03:02:39.066458 2633 tablet_replica.cc:439] T 359a5d64e02445eaafa86a640015964a P 6b4bf12bbdea4f47884ccb78a5476f45: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e7456a33b6a46d5bf984827a286a2f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } } }
I20260924 03:02:39.066534 2633 tablet_replica.cc:442] T 359a5d64e02445eaafa86a640015964a P 6b4bf12bbdea4f47884ccb78a5476f45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:02:39.066919 2651 tablet_replica.cc:439] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e7456a33b6a46d5bf984827a286a2f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } } }
I20260924 03:02:39.066994 2651 tablet_replica.cc:442] T 359a5d64e02445eaafa86a640015964a P 5e7456a33b6a46d5bf984827a286a2f7: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:02:39.067247 2632 tablet_replica.cc:439] T 359a5d64e02445eaafa86a640015964a P d97f3824f995462e83f56ff3c27bf690: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e7456a33b6a46d5bf984827a286a2f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } } }
I20260924 03:02:39.067329 2632 tablet_replica.cc:442] T 359a5d64e02445eaafa86a640015964a P d97f3824f995462e83f56ff3c27bf690: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:02:39.067514 2657 mvcc.cc:205] Tried to move back new op lower bound from 7332736856320823296 to 7332736856185384960. Current Snapshot: MvccSnapshot[applied={T|T < 7332736856320823296}]
I20260924 03:02:39.067623 2658 mvcc.cc:205] Tried to move back new op lower bound from 7332736856320823296 to 7332736856185384960. Current Snapshot: MvccSnapshot[applied={T|T < 7332736856320823296}]
I20260924 03:02:39.072418 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:39.069514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43504 (local address 127.0.98.2:35617)
0924 03:02:39.069556 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:39.069558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:39.069731 (+ 173us) server_negotiation.cc:407] Connection header received
0924 03:02:39.069794 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:39.069795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:39.069810 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:39.069861 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:02:39.070004 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.070542 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.071691 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.071885 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.071938 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:39.072108 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:02:39.072197 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:02:39.072253 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:39.072318 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 03:02:39.072337 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:02:39.077764 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:39.075184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53953 (local address 127.0.98.1:36911)
0924 03:02:39.075235 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:39.075237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:39.075378 (+ 141us) server_negotiation.cc:407] Connection header received
0924 03:02:39.075467 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:39.075467 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:39.075491 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:39.075574 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:02:39.075738 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.076200 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.077016 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.077219 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.077306 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:39.077502 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:02:39.077543 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:02:39.077606 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:39.077669 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 03:02:39.077689 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 03:02:39.077806 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:39.075117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36911 (local address 127.0.98.2:53953)
0924 03:02:39.075298 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:39.075309 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:39.075391 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:39.075579 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:39.075580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:39.075581 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:02:39.075669 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:39.075670 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.076319 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.076320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:39.076906 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:39.076909 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.077307 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.077308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:39.077374 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:39.077390 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:02:39.077607 (+ 217us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:02:39.077609 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:02:39.077667 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 03:02:39.077710 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":93,"threads_started":1}
I20260924 03:02:39.080044 2557 raft_consensus.cc:1261] T 6dc7692ff4b740528e95f6f231e550bc P d97f3824f995462e83f56ff3c27bf690 [term 1 FOLLOWER]: Refusing update from remote peer 6b4bf12bbdea4f47884ccb78a5476f45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:02:39.080152 2277 raft_consensus.cc:1261] T 6dc7692ff4b740528e95f6f231e550bc P 5e7456a33b6a46d5bf984827a286a2f7 [term 1 FOLLOWER]: Refusing update from remote peer 6b4bf12bbdea4f47884ccb78a5476f45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:02:39.080299 2633 consensus_queue.cc:1035] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:02:39.080422 2637 consensus_queue.cc:1035] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:02:39.089329 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:39.086646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39859 (local address 127.0.98.2:35617)
0924 03:02:39.086692 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:39.086694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:39.086850 (+ 156us) server_negotiation.cc:407] Connection header received
0924 03:02:39.086908 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:39.086908 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:39.086916 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:39.086960 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:02:39.087071 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.087513 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.088667 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.088924 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.088973 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:39.089151 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:02:39.089176 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:02:39.089208 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:39.089260 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 03:02:39.089276 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 03:02:39.089393 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:39.086615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35617 (local address 127.0.98.1:39859)
0924 03:02:39.086788 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:39.086800 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:39.086860 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:39.086965 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:39.086966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:39.086967 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:02:39.087017 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:39.087018 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.087580 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.087581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:39.088589 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:39.088592 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.088985 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.088986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:39.089046 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:39.089062 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:02:39.089210 (+ 148us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:02:39.089211 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 03:02:39.089259 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 03:02:39.089302 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":93,"threads_started":1}
I20260924 03:02:39.098496 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:02:39.098631 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:02:39.099460 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a20d6abbc75447a69eb1478b7b10f2f2: Sending DeleteTablet for 3 replicas of tablet 6dc7692ff4b740528e95f6f231e550bc
I20260924 03:02:39.099812 2257 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dc7692ff4b740528e95f6f231e550bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:02:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45510
I20260924 03:02:39.099912 2537 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dc7692ff4b740528e95f6f231e550bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:02:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39402
I20260924 03:02:39.099934 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dc7692ff4b740528e95f6f231e550bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:02:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43498
I20260924 03:02:39.100319 2679 tablet_replica.cc:330] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45: stopping tablet replica
I20260924 03:02:39.100427 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:02:39.100584 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:02:39.100893 2679 raft_consensus.cc:2229] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:02:39.101156 2679 raft_consensus.cc:2258] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:02:39.101948 2537 tablet_service.cc:1467] Processing CreateTablet for tablet e2194964108a4fb9aca0dd650522a855 (DEFAULT_TABLE table=test-table [id=8a8c953da98b434fb54da7c6aab9a665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:39.101956 2397 tablet_service.cc:1467] Processing CreateTablet for tablet e2194964108a4fb9aca0dd650522a855 (DEFAULT_TABLE table=test-table [id=8a8c953da98b434fb54da7c6aab9a665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:39.102082 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2194964108a4fb9aca0dd650522a855. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:39.102082 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2194964108a4fb9aca0dd650522a855. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:39.103479 2628 tablet_bootstrap.cc:492] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690: Bootstrap starting.
I20260924 03:02:39.103484 2627 tablet_bootstrap.cc:492] T e2194964108a4fb9aca0dd650522a855 P 6b4bf12bbdea4f47884ccb78a5476f45: Bootstrap starting.
I20260924 03:02:39.103942 2628 tablet_bootstrap.cc:654] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:39.104048 2627 tablet_bootstrap.cc:654] T e2194964108a4fb9aca0dd650522a855 P 6b4bf12bbdea4f47884ccb78a5476f45: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:39.104557 2257 tablet_service.cc:1467] Processing CreateTablet for tablet e2194964108a4fb9aca0dd650522a855 (DEFAULT_TABLE table=test-table [id=8a8c953da98b434fb54da7c6aab9a665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:39.104710 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2194964108a4fb9aca0dd650522a855. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:39.104956 2628 tablet_bootstrap.cc:492] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690: No bootstrap required, opened a new log
I20260924 03:02:39.105001 2628 ts_tablet_manager.cc:1397] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 03:02:39.105159 2628 raft_consensus.cc:348] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:39.105219 2628 raft_consensus.cc:374] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:39.105237 2628 raft_consensus.cc:729] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d97f3824f995462e83f56ff3c27bf690, State: Initialized, Role: FOLLOWER
I20260924 03:02:39.105276 2628 consensus_queue.cc:260] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:39.105351 2628 ts_tablet_manager.cc:1428] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:39.106011 2626 tablet_bootstrap.cc:492] T e2194964108a4fb9aca0dd650522a855 P 5e7456a33b6a46d5bf984827a286a2f7: Bootstrap starting.
I20260924 03:02:39.106338 2626 tablet_bootstrap.cc:654] T e2194964108a4fb9aca0dd650522a855 P 5e7456a33b6a46d5bf984827a286a2f7: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:39.106942 2679 ts_tablet_manager.cc:1905] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:02:39.107959 2626 tablet_bootstrap.cc:492] T e2194964108a4fb9aca0dd650522a855 P 5e7456a33b6a46d5bf984827a286a2f7: No bootstrap required, opened a new log
I20260924 03:02:39.108007 2626 ts_tablet_manager.cc:1397] T e2194964108a4fb9aca0dd650522a855 P 5e7456a33b6a46d5bf984827a286a2f7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:02:39.108152 2626 raft_consensus.cc:348] T e2194964108a4fb9aca0dd650522a855 P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:39.108211 2626 raft_consensus.cc:374] T e2194964108a4fb9aca0dd650522a855 P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:39.108229 2626 raft_consensus.cc:729] T e2194964108a4fb9aca0dd650522a855 P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7456a33b6a46d5bf984827a286a2f7, State: Initialized, Role: FOLLOWER
I20260924 03:02:39.108280 2626 consensus_queue.cc:260] T e2194964108a4fb9aca0dd650522a855 P 5e7456a33b6a46d5bf984827a286a2f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:39.108366 2626 ts_tablet_manager.cc:1428] T e2194964108a4fb9aca0dd650522a855 P 5e7456a33b6a46d5bf984827a286a2f7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:39.108476 2677 tablet_replica.cc:330] T 6dc7692ff4b740528e95f6f231e550bc P 5e7456a33b6a46d5bf984827a286a2f7: stopping tablet replica
I20260924 03:02:39.108594 2677 raft_consensus.cc:2229] T 6dc7692ff4b740528e95f6f231e550bc P 5e7456a33b6a46d5bf984827a286a2f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:02:39.108649 2677 raft_consensus.cc:2258] T 6dc7692ff4b740528e95f6f231e550bc P 5e7456a33b6a46d5bf984827a286a2f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:02:39.108803 2678 tablet_replica.cc:330] T 6dc7692ff4b740528e95f6f231e550bc P d97f3824f995462e83f56ff3c27bf690: stopping tablet replica
I20260924 03:02:39.108920 2678 raft_consensus.cc:2229] T 6dc7692ff4b740528e95f6f231e550bc P d97f3824f995462e83f56ff3c27bf690 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:02:39.108985 2678 raft_consensus.cc:2258] T 6dc7692ff4b740528e95f6f231e550bc P d97f3824f995462e83f56ff3c27bf690 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:02:39.109198 2627 tablet_bootstrap.cc:492] T e2194964108a4fb9aca0dd650522a855 P 6b4bf12bbdea4f47884ccb78a5476f45: No bootstrap required, opened a new log
I20260924 03:02:39.109257 2627 ts_tablet_manager.cc:1397] T e2194964108a4fb9aca0dd650522a855 P 6b4bf12bbdea4f47884ccb78a5476f45: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260924 03:02:39.109414 2627 raft_consensus.cc:348] T e2194964108a4fb9aca0dd650522a855 P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:39.109500 2627 raft_consensus.cc:374] T e2194964108a4fb9aca0dd650522a855 P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:39.109525 2627 raft_consensus.cc:729] T e2194964108a4fb9aca0dd650522a855 P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b4bf12bbdea4f47884ccb78a5476f45, State: Initialized, Role: FOLLOWER
I20260924 03:02:39.109579 2627 consensus_queue.cc:260] T e2194964108a4fb9aca0dd650522a855 P 6b4bf12bbdea4f47884ccb78a5476f45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:39.109701 2627 ts_tablet_manager.cc:1428] T e2194964108a4fb9aca0dd650522a855 P 6b4bf12bbdea4f47884ccb78a5476f45: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 03:02:39.109761 2679 ts_tablet_manager.cc:1918] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:02:39.109831 2679 log.cc:1196] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-1/wal/wals/6dc7692ff4b740528e95f6f231e550bc
I20260924 03:02:39.110111 2679 ts_tablet_manager.cc:1939] T 6dc7692ff4b740528e95f6f231e550bc P 6b4bf12bbdea4f47884ccb78a5476f45: Deleting consensus metadata
I20260924 03:02:39.110898 2677 ts_tablet_manager.cc:1905] T 6dc7692ff4b740528e95f6f231e550bc P 5e7456a33b6a46d5bf984827a286a2f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:02:39.111157 2678 ts_tablet_manager.cc:1905] T 6dc7692ff4b740528e95f6f231e550bc P d97f3824f995462e83f56ff3c27bf690: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:02:39.114610 2678 ts_tablet_manager.cc:1918] T 6dc7692ff4b740528e95f6f231e550bc P d97f3824f995462e83f56ff3c27bf690: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:02:39.114696 2678 log.cc:1196] T 6dc7692ff4b740528e95f6f231e550bc P d97f3824f995462e83f56ff3c27bf690: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-2/wal/wals/6dc7692ff4b740528e95f6f231e550bc
I20260924 03:02:39.114980 2678 ts_tablet_manager.cc:1939] T 6dc7692ff4b740528e95f6f231e550bc P d97f3824f995462e83f56ff3c27bf690: Deleting consensus metadata
I20260924 03:02:39.115433 2677 ts_tablet_manager.cc:1918] T 6dc7692ff4b740528e95f6f231e550bc P 5e7456a33b6a46d5bf984827a286a2f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:02:39.115422 2106 catalog_manager.cc:4841] TS d97f3824f995462e83f56ff3c27bf690 (127.0.98.3:39243): tablet 6dc7692ff4b740528e95f6f231e550bc (table test-table [id=f52e653e194f418a8f1d25896ef1736b]) successfully deleted
I20260924 03:02:39.115511 2677 log.cc:1196] T 6dc7692ff4b740528e95f6f231e550bc P 5e7456a33b6a46d5bf984827a286a2f7: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-0/wal/wals/6dc7692ff4b740528e95f6f231e550bc
I20260924 03:02:39.115710 2106 catalog_manager.cc:4841] TS 6b4bf12bbdea4f47884ccb78a5476f45 (127.0.98.2:35617): tablet 6dc7692ff4b740528e95f6f231e550bc (table test-table [id=f52e653e194f418a8f1d25896ef1736b]) successfully deleted
I20260924 03:02:39.115816 2677 ts_tablet_manager.cc:1939] T 6dc7692ff4b740528e95f6f231e550bc P 5e7456a33b6a46d5bf984827a286a2f7: Deleting consensus metadata
I20260924 03:02:39.116227 2105 catalog_manager.cc:4841] TS 5e7456a33b6a46d5bf984827a286a2f7 (127.0.98.1:36911): tablet 6dc7692ff4b740528e95f6f231e550bc (table test-table [id=f52e653e194f418a8f1d25896ef1736b]) successfully deleted
I20260924 03:02:39.372460 2632 raft_consensus.cc:482] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:02:39.372574 2632 raft_consensus.cc:504] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:39.373008 2632 leader_election.cc:288] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e7456a33b6a46d5bf984827a286a2f7 (127.0.98.1:36911), 6b4bf12bbdea4f47884ccb78a5476f45 (127.0.98.2:35617)
I20260924 03:02:39.377586 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:39.373496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56931 (local address 127.0.98.1:36911)
0924 03:02:39.373557 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:39.373559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:39.373567 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:02:39.373599 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:39.373599 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:39.373610 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:39.373705 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:39.374025 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.374782 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.376573 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.377123 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.377209 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:39.377222 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:39.377424 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 03:02:39.377468 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 03:02:39.377615 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:39.373240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35617 (local address 127.0.98.3:41719)
0924 03:02:39.373469 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:39.373478 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:39.373535 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:39.373777 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:39.373778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:39.373781 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:39.373908 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:39.373911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.374900 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.374901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:39.376398 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:39.376401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.377313 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.377314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:39.377393 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:39.377397 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:02:39.377487 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 03:02:39.377518 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":5,"spinlock_wait_cycles":768,"thread_start_us":140,"threads_started":1}
I20260924 03:02:39.377668 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:39.373424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41719 (local address 127.0.98.2:35617)
0924 03:02:39.373478 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:39.373480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:39.373542 (+ 62us) server_negotiation.cc:407] Connection header received
0924 03:02:39.373599 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:39.373623 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:39.373634 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:39.373730 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:39.374113 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.374817 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.376574 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.377216 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.377282 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:39.377291 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:39.377536 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 03:02:39.377579 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 03:02:39.377624 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:39.373240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36911 (local address 127.0.98.3:56931)
0924 03:02:39.373404 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:39.373417 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:39.373518 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:39.373881 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:39.373883 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:39.373884 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:39.373945 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:39.373946 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.374898 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.374899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:39.376399 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:39.376401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.377222 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.377223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:39.377334 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:39.377339 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:02:39.377409 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 03:02:39.377496 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":5,"thread_start_us":74,"threads_started":1}
I20260924 03:02:39.377882 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2194964108a4fb9aca0dd650522a855" candidate_uuid: "d97f3824f995462e83f56ff3c27bf690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" is_pre_election: true
I20260924 03:02:39.377990 2417 raft_consensus.cc:2454] T e2194964108a4fb9aca0dd650522a855 P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d97f3824f995462e83f56ff3c27bf690 in term 0.
I20260924 03:02:39.378142 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2194964108a4fb9aca0dd650522a855" candidate_uuid: "d97f3824f995462e83f56ff3c27bf690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e7456a33b6a46d5bf984827a286a2f7" is_pre_election: true
I20260924 03:02:39.378185 2490 leader_election.cc:302] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b4bf12bbdea4f47884ccb78a5476f45, d97f3824f995462e83f56ff3c27bf690; no voters:
I20260924 03:02:39.378242 2277 raft_consensus.cc:2454] T e2194964108a4fb9aca0dd650522a855 P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d97f3824f995462e83f56ff3c27bf690 in term 0.
I20260924 03:02:39.378315 2632 raft_consensus.cc:2781] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:02:39.378374 2632 raft_consensus.cc:482] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:02:39.378401 2632 raft_consensus.cc:3037] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:39.379132 2632 raft_consensus.cc:504] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:39.379273 2632 leader_election.cc:288] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690 [CANDIDATE]: Term 1 election: Requested vote from peers 5e7456a33b6a46d5bf984827a286a2f7 (127.0.98.1:36911), 6b4bf12bbdea4f47884ccb78a5476f45 (127.0.98.2:35617)
I20260924 03:02:39.379451 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2194964108a4fb9aca0dd650522a855" candidate_uuid: "d97f3824f995462e83f56ff3c27bf690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b4bf12bbdea4f47884ccb78a5476f45"
I20260924 03:02:39.379436 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2194964108a4fb9aca0dd650522a855" candidate_uuid: "d97f3824f995462e83f56ff3c27bf690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e7456a33b6a46d5bf984827a286a2f7"
I20260924 03:02:39.379528 2417 raft_consensus.cc:3037] T e2194964108a4fb9aca0dd650522a855 P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:39.379558 2277 raft_consensus.cc:3037] T e2194964108a4fb9aca0dd650522a855 P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:39.380365 2277 raft_consensus.cc:2454] T e2194964108a4fb9aca0dd650522a855 P 5e7456a33b6a46d5bf984827a286a2f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d97f3824f995462e83f56ff3c27bf690 in term 1.
I20260924 03:02:39.380460 2417 raft_consensus.cc:2454] T e2194964108a4fb9aca0dd650522a855 P 6b4bf12bbdea4f47884ccb78a5476f45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d97f3824f995462e83f56ff3c27bf690 in term 1.
I20260924 03:02:39.380605 2489 leader_election.cc:302] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e7456a33b6a46d5bf984827a286a2f7, d97f3824f995462e83f56ff3c27bf690; no voters:
I20260924 03:02:39.380764 2632 raft_consensus.cc:2781] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:39.380918 2632 raft_consensus.cc:686] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690 [term 1 LEADER]: Becoming Leader. State: Replica: d97f3824f995462e83f56ff3c27bf690, State: Running, Role: LEADER
I20260924 03:02:39.381002 2632 consensus_queue.cc:237] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:39.381879 2120 catalog_manager.cc:5495] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690 reported cstate change: term changed from 0 to 1, leader changed from <none> to d97f3824f995462e83f56ff3c27bf690 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d97f3824f995462e83f56ff3c27bf690" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } health_report { overall_health: HEALTHY } } }
I20260924 03:02:39.394136 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:39.391676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39408 (local address 127.0.98.3:39243)
0924 03:02:39.391717 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:39.391719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:39.391770 (+ 51us) server_negotiation.cc:407] Connection header received
0924 03:02:39.391837 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:39.391838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:39.391854 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:39.391919 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:02:39.391996 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.392457 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.393303 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.393481 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.393539 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:39.393773 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:02:39.393897 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:02:39.393954 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:39.393999 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 03:02:39.394016 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:02:39.399355 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:39.396859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46179 (local address 127.0.98.1:36911)
0924 03:02:39.396923 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:39.396924 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:39.397006 (+ 82us) server_negotiation.cc:407] Connection header received
0924 03:02:39.397065 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:39.397066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:39.397074 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:39.397121 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:02:39.397493 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.397936 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.398755 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.398927 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.398979 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:39.399172 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:02:39.399195 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:02:39.399232 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:39.399285 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 03:02:39.399299 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 03:02:39.399385 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:39.396782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36911 (local address 127.0.98.3:46179)
0924 03:02:39.396929 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:39.396943 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:39.397007 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:39.397122 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:39.397122 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:39.397124 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:02:39.397439 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:39.397440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.398004 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.398004 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:39.398682 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:39.398684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.399026 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.399027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:39.399079 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:39.399092 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:02:39.399248 (+ 156us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:02:39.399249 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 03:02:39.399285 (+ 36us) client_negotiation.cc:240] Negotiation successful
0924 03:02:39.399318 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260924 03:02:39.401662 2277 raft_consensus.cc:1261] T e2194964108a4fb9aca0dd650522a855 P 5e7456a33b6a46d5bf984827a286a2f7 [term 1 FOLLOWER]: Refusing update from remote peer d97f3824f995462e83f56ff3c27bf690: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:02:39.401824 2417 raft_consensus.cc:1261] T e2194964108a4fb9aca0dd650522a855 P 6b4bf12bbdea4f47884ccb78a5476f45 [term 1 FOLLOWER]: Refusing update from remote peer d97f3824f995462e83f56ff3c27bf690: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:02:39.401947 2632 consensus_queue.cc:1035] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:02:39.402331 2632 consensus_queue.cc:1035] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:02:39.409600 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:39.407062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53909 (local address 127.0.98.3:39243)
0924 03:02:39.407104 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:39.407105 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:39.407113 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:02:39.407133 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:39.407134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:39.407144 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:39.407210 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:02:39.407353 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.407966 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.408766 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.409038 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.409115 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:39.409271 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:02:39.409306 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:02:39.409359 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:39.409488 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 03:02:39.409540 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:02:39.409607 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:39.406930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39243 (local address 127.0.98.1:53909)
0924 03:02:39.406980 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:39.406991 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:39.407029 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:39.407235 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:39.407236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:39.407238 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:02:39.407282 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:39.407284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.408037 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.408038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:39.408583 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:39.408585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:39.409133 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:39.409134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:39.409188 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:39.409196 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:02:39.409422 (+ 226us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:02:39.409424 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:02:39.409483 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 03:02:39.409535 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 03:02:39.409960 2658 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 03:02:40.929545 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:02:40.929646 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:02:40.930536 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a20d6abbc75447a69eb1478b7b10f2f2: Sending DeleteTablet for 3 replicas of tablet e2194964108a4fb9aca0dd650522a855
I20260924 03:02:40.930794 2257 tablet_service.cc:1514] Processing DeleteTablet for tablet e2194964108a4fb9aca0dd650522a855 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:02:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45510
I20260924 03:02:40.930857 2537 tablet_service.cc:1514] Processing DeleteTablet for tablet e2194964108a4fb9aca0dd650522a855 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:02:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39402
I20260924 03:02:40.930907 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet e2194964108a4fb9aca0dd650522a855 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:02:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43498
I20260924 03:02:40.931111 2714 tablet_replica.cc:330] T e2194964108a4fb9aca0dd650522a855 P 5e7456a33b6a46d5bf984827a286a2f7: stopping tablet replica
I20260924 03:02:40.931172 2714 raft_consensus.cc:2229] T e2194964108a4fb9aca0dd650522a855 P 5e7456a33b6a46d5bf984827a286a2f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:02:40.931164 2715 tablet_replica.cc:330] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690: stopping tablet replica
I20260924 03:02:40.931216 2714 raft_consensus.cc:2258] T e2194964108a4fb9aca0dd650522a855 P 5e7456a33b6a46d5bf984827a286a2f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:02:40.931223 2715 raft_consensus.cc:2229] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:02:40.931329 2715 raft_consensus.cc:2258] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:02:40.931437 2716 tablet_replica.cc:330] T e2194964108a4fb9aca0dd650522a855 P 6b4bf12bbdea4f47884ccb78a5476f45: stopping tablet replica
I20260924 03:02:40.931464 2714 ts_tablet_manager.cc:1905] T e2194964108a4fb9aca0dd650522a855 P 5e7456a33b6a46d5bf984827a286a2f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:02:40.931486 2716 raft_consensus.cc:2229] T e2194964108a4fb9aca0dd650522a855 P 6b4bf12bbdea4f47884ccb78a5476f45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:02:40.931525 2716 raft_consensus.cc:2258] T e2194964108a4fb9aca0dd650522a855 P 6b4bf12bbdea4f47884ccb78a5476f45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:02:40.931731 2715 ts_tablet_manager.cc:1905] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:02:40.933290 2715 ts_tablet_manager.cc:1918] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 03:02:40.933342 2715 log.cc:1196] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-2/wal/wals/e2194964108a4fb9aca0dd650522a855
I20260924 03:02:40.933454 2714 ts_tablet_manager.cc:1918] T e2194964108a4fb9aca0dd650522a855 P 5e7456a33b6a46d5bf984827a286a2f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 03:02:40.933514 2714 log.cc:1196] T e2194964108a4fb9aca0dd650522a855 P 5e7456a33b6a46d5bf984827a286a2f7: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-0/wal/wals/e2194964108a4fb9aca0dd650522a855
I20260924 03:02:40.933584 2715 ts_tablet_manager.cc:1939] T e2194964108a4fb9aca0dd650522a855 P d97f3824f995462e83f56ff3c27bf690: Deleting consensus metadata
I20260924 03:02:40.933766 2714 ts_tablet_manager.cc:1939] T e2194964108a4fb9aca0dd650522a855 P 5e7456a33b6a46d5bf984827a286a2f7: Deleting consensus metadata
I20260924 03:02:40.933871 2716 ts_tablet_manager.cc:1905] T e2194964108a4fb9aca0dd650522a855 P 6b4bf12bbdea4f47884ccb78a5476f45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:02:40.934144 2105 catalog_manager.cc:4841] TS 5e7456a33b6a46d5bf984827a286a2f7 (127.0.98.1:36911): tablet e2194964108a4fb9aca0dd650522a855 (table test-table [id=8a8c953da98b434fb54da7c6aab9a665]) successfully deleted
I20260924 03:02:40.935326 2106 catalog_manager.cc:4841] TS d97f3824f995462e83f56ff3c27bf690 (127.0.98.3:39243): tablet e2194964108a4fb9aca0dd650522a855 (table test-table [id=8a8c953da98b434fb54da7c6aab9a665]) successfully deleted
I20260924 03:02:40.935521 2716 ts_tablet_manager.cc:1918] T e2194964108a4fb9aca0dd650522a855 P 6b4bf12bbdea4f47884ccb78a5476f45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 03:02:40.935642 2716 log.cc:1196] T e2194964108a4fb9aca0dd650522a855 P 6b4bf12bbdea4f47884ccb78a5476f45: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-1/wal/wals/e2194964108a4fb9aca0dd650522a855
I20260924 03:02:40.935855 2716 ts_tablet_manager.cc:1939] T e2194964108a4fb9aca0dd650522a855 P 6b4bf12bbdea4f47884ccb78a5476f45: Deleting consensus metadata
I20260924 03:02:40.936208 2106 catalog_manager.cc:4841] TS 6b4bf12bbdea4f47884ccb78a5476f45 (127.0.98.2:35617): tablet e2194964108a4fb9aca0dd650522a855 (table test-table [id=8a8c953da98b434fb54da7c6aab9a665]) successfully deleted
Sep 24 03:02:40 dist-test-slave-wgsb krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218960, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 03:02:40 dist-test-slave-wgsb krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218960, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:40.957479 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:40.949358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59890 (local address 127.0.98.62:33383)
0924 03:02:40.949537 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:40.949539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:40.949614 (+ 75us) server_negotiation.cc:407] Connection header received
0924 03:02:40.949720 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:40.949721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:40.949734 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:40.949817 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:40.950487 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:40.951007 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:40.951938 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:40.952158 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:40.952243 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:40.954655 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:40.954657 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:40.954657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:40.954664 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:40.956298 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:40.956663 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:40.956665 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:40.956665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:40.956731 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:40.956941 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:40.956942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:40.956942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:40.957058 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:40.957161 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:40.957284 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 03:02:40.957364 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":80,"threads_started":1}
I20260924 03:02:40.959019 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:02:40.959167 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:02:40.960620 2257 tablet_service.cc:1467] Processing CreateTablet for tablet c7ee8f364a044d7abe18ed980f86aab9 (DEFAULT_TABLE table=test-table [id=0fb9fc4545f04deaa474d1d98c98ad1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:40.960650 2397 tablet_service.cc:1467] Processing CreateTablet for tablet c7ee8f364a044d7abe18ed980f86aab9 (DEFAULT_TABLE table=test-table [id=0fb9fc4545f04deaa474d1d98c98ad1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:40.960724 2537 tablet_service.cc:1467] Processing CreateTablet for tablet c7ee8f364a044d7abe18ed980f86aab9 (DEFAULT_TABLE table=test-table [id=0fb9fc4545f04deaa474d1d98c98ad1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:40.960770 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7ee8f364a044d7abe18ed980f86aab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:40.960784 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7ee8f364a044d7abe18ed980f86aab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:40.960848 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7ee8f364a044d7abe18ed980f86aab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:40.962069 2725 tablet_bootstrap.cc:492] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45: Bootstrap starting.
I20260924 03:02:40.962312 2726 tablet_bootstrap.cc:492] T c7ee8f364a044d7abe18ed980f86aab9 P 5e7456a33b6a46d5bf984827a286a2f7: Bootstrap starting.
I20260924 03:02:40.962488 2725 tablet_bootstrap.cc:654] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:40.962592 2726 tablet_bootstrap.cc:654] T c7ee8f364a044d7abe18ed980f86aab9 P 5e7456a33b6a46d5bf984827a286a2f7: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:40.962881 2727 tablet_bootstrap.cc:492] T c7ee8f364a044d7abe18ed980f86aab9 P d97f3824f995462e83f56ff3c27bf690: Bootstrap starting.
I20260924 03:02:40.963169 2725 tablet_bootstrap.cc:492] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45: No bootstrap required, opened a new log
I20260924 03:02:40.963208 2727 tablet_bootstrap.cc:654] T c7ee8f364a044d7abe18ed980f86aab9 P d97f3824f995462e83f56ff3c27bf690: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:40.963236 2725 ts_tablet_manager.cc:1397] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:40.963363 2725 raft_consensus.cc:348] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:40.963440 2725 raft_consensus.cc:374] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:40.963469 2725 raft_consensus.cc:729] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b4bf12bbdea4f47884ccb78a5476f45, State: Initialized, Role: FOLLOWER
I20260924 03:02:40.963512 2725 consensus_queue.cc:260] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:40.963732 2725 ts_tablet_manager.cc:1428] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 03:02:40.963825 2726 tablet_bootstrap.cc:492] T c7ee8f364a044d7abe18ed980f86aab9 P 5e7456a33b6a46d5bf984827a286a2f7: No bootstrap required, opened a new log
I20260924 03:02:40.963883 2726 ts_tablet_manager.cc:1397] T c7ee8f364a044d7abe18ed980f86aab9 P 5e7456a33b6a46d5bf984827a286a2f7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:02:40.963905 2727 tablet_bootstrap.cc:492] T c7ee8f364a044d7abe18ed980f86aab9 P d97f3824f995462e83f56ff3c27bf690: No bootstrap required, opened a new log
I20260924 03:02:40.963960 2727 ts_tablet_manager.cc:1397] T c7ee8f364a044d7abe18ed980f86aab9 P d97f3824f995462e83f56ff3c27bf690: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:40.963997 2726 raft_consensus.cc:348] T c7ee8f364a044d7abe18ed980f86aab9 P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:40.964087 2726 raft_consensus.cc:374] T c7ee8f364a044d7abe18ed980f86aab9 P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:40.964073 2727 raft_consensus.cc:348] T c7ee8f364a044d7abe18ed980f86aab9 P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:40.964143 2727 raft_consensus.cc:374] T c7ee8f364a044d7abe18ed980f86aab9 P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:40.964131 2726 raft_consensus.cc:729] T c7ee8f364a044d7abe18ed980f86aab9 P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7456a33b6a46d5bf984827a286a2f7, State: Initialized, Role: FOLLOWER
I20260924 03:02:40.964171 2727 raft_consensus.cc:729] T c7ee8f364a044d7abe18ed980f86aab9 P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d97f3824f995462e83f56ff3c27bf690, State: Initialized, Role: FOLLOWER
I20260924 03:02:40.964216 2726 consensus_queue.cc:260] T c7ee8f364a044d7abe18ed980f86aab9 P 5e7456a33b6a46d5bf984827a286a2f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:40.964228 2727 consensus_queue.cc:260] T c7ee8f364a044d7abe18ed980f86aab9 P d97f3824f995462e83f56ff3c27bf690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:40.964353 2726 ts_tablet_manager.cc:1428] T c7ee8f364a044d7abe18ed980f86aab9 P 5e7456a33b6a46d5bf984827a286a2f7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:40.964532 2727 ts_tablet_manager.cc:1428] T c7ee8f364a044d7abe18ed980f86aab9 P d97f3824f995462e83f56ff3c27bf690: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 03:02:41.218418 2731 raft_consensus.cc:482] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:02:41.218529 2731 raft_consensus.cc:504] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:41.218729 2731 leader_election.cc:288] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e7456a33b6a46d5bf984827a286a2f7 (127.0.98.1:36911), d97f3824f995462e83f56ff3c27bf690 (127.0.98.3:39243)
I20260924 03:02:41.218982 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7ee8f364a044d7abe18ed980f86aab9" candidate_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d97f3824f995462e83f56ff3c27bf690" is_pre_election: true
I20260924 03:02:41.218993 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7ee8f364a044d7abe18ed980f86aab9" candidate_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e7456a33b6a46d5bf984827a286a2f7" is_pre_election: true
I20260924 03:02:41.219098 2557 raft_consensus.cc:2454] T c7ee8f364a044d7abe18ed980f86aab9 P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b4bf12bbdea4f47884ccb78a5476f45 in term 0.
I20260924 03:02:41.219091 2277 raft_consensus.cc:2454] T c7ee8f364a044d7abe18ed980f86aab9 P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b4bf12bbdea4f47884ccb78a5476f45 in term 0.
I20260924 03:02:41.219275 2350 leader_election.cc:302] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b4bf12bbdea4f47884ccb78a5476f45, d97f3824f995462e83f56ff3c27bf690; no voters:
I20260924 03:02:41.219403 2731 raft_consensus.cc:2781] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:02:41.219457 2731 raft_consensus.cc:482] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:02:41.219484 2731 raft_consensus.cc:3037] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:41.220364 2731 raft_consensus.cc:504] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:41.220583 2731 leader_election.cc:288] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45 [CANDIDATE]: Term 1 election: Requested vote from peers 5e7456a33b6a46d5bf984827a286a2f7 (127.0.98.1:36911), d97f3824f995462e83f56ff3c27bf690 (127.0.98.3:39243)
I20260924 03:02:41.220677 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7ee8f364a044d7abe18ed980f86aab9" candidate_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d97f3824f995462e83f56ff3c27bf690"
I20260924 03:02:41.220767 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7ee8f364a044d7abe18ed980f86aab9" candidate_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e7456a33b6a46d5bf984827a286a2f7"
I20260924 03:02:41.220798 2557 raft_consensus.cc:3037] T c7ee8f364a044d7abe18ed980f86aab9 P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:41.220847 2277 raft_consensus.cc:3037] T c7ee8f364a044d7abe18ed980f86aab9 P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:41.221522 2557 raft_consensus.cc:2454] T c7ee8f364a044d7abe18ed980f86aab9 P d97f3824f995462e83f56ff3c27bf690 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b4bf12bbdea4f47884ccb78a5476f45 in term 1.
I20260924 03:02:41.221526 2277 raft_consensus.cc:2454] T c7ee8f364a044d7abe18ed980f86aab9 P 5e7456a33b6a46d5bf984827a286a2f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b4bf12bbdea4f47884ccb78a5476f45 in term 1.
I20260924 03:02:41.221724 2350 leader_election.cc:302] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b4bf12bbdea4f47884ccb78a5476f45, d97f3824f995462e83f56ff3c27bf690; no voters:
I20260924 03:02:41.221817 2731 raft_consensus.cc:2781] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:41.221977 2731 raft_consensus.cc:686] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45 [term 1 LEADER]: Becoming Leader. State: Replica: 6b4bf12bbdea4f47884ccb78a5476f45, State: Running, Role: LEADER
I20260924 03:02:41.222023 2731 consensus_queue.cc:237] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:41.222658 2118 catalog_manager.cc:5495] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b4bf12bbdea4f47884ccb78a5476f45 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } health_report { overall_health: UNKNOWN } } }
I20260924 03:02:41.251194 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:41.248681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53866 (local address 127.0.98.2:35617)
0924 03:02:41.248836 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:41.248837 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:41.248845 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:02:41.248872 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:41.248872 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:41.248881 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:41.248954 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:02:41.249084 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:41.249635 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:41.250415 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:41.250630 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:41.250715 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:41.250890 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:02:41.250931 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:02:41.251003 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:41.251104 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 03:02:41.251127 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":102,"threads_started":1}
I20260924 03:02:41.253901 2277 raft_consensus.cc:1261] T c7ee8f364a044d7abe18ed980f86aab9 P 5e7456a33b6a46d5bf984827a286a2f7 [term 1 FOLLOWER]: Refusing update from remote peer 6b4bf12bbdea4f47884ccb78a5476f45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:02:41.254035 2557 raft_consensus.cc:1261] T c7ee8f364a044d7abe18ed980f86aab9 P d97f3824f995462e83f56ff3c27bf690 [term 1 FOLLOWER]: Refusing update from remote peer 6b4bf12bbdea4f47884ccb78a5476f45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:02:41.254142 2732 consensus_queue.cc:1035] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:02:41.254278 2731 consensus_queue.cc:1035] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:02:41.264477 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:02:41.264593 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:02:41.265232 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a20d6abbc75447a69eb1478b7b10f2f2: Sending DeleteTablet for 3 replicas of tablet c7ee8f364a044d7abe18ed980f86aab9
I20260924 03:02:41.265561 2257 tablet_service.cc:1514] Processing DeleteTablet for tablet c7ee8f364a044d7abe18ed980f86aab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:02:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45510
I20260924 03:02:41.265614 2537 tablet_service.cc:1514] Processing DeleteTablet for tablet c7ee8f364a044d7abe18ed980f86aab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:02:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39402
I20260924 03:02:41.265734 2714 tablet_replica.cc:330] T c7ee8f364a044d7abe18ed980f86aab9 P 5e7456a33b6a46d5bf984827a286a2f7: stopping tablet replica
I20260924 03:02:41.265739 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet c7ee8f364a044d7abe18ed980f86aab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:02:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43498
I20260924 03:02:41.265758 2715 tablet_replica.cc:330] T c7ee8f364a044d7abe18ed980f86aab9 P d97f3824f995462e83f56ff3c27bf690: stopping tablet replica
I20260924 03:02:41.265815 2714 raft_consensus.cc:2229] T c7ee8f364a044d7abe18ed980f86aab9 P 5e7456a33b6a46d5bf984827a286a2f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:02:41.265859 2715 raft_consensus.cc:2229] T c7ee8f364a044d7abe18ed980f86aab9 P d97f3824f995462e83f56ff3c27bf690 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:02:41.265865 2714 raft_consensus.cc:2258] T c7ee8f364a044d7abe18ed980f86aab9 P 5e7456a33b6a46d5bf984827a286a2f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:02:41.265906 2715 raft_consensus.cc:2258] T c7ee8f364a044d7abe18ed980f86aab9 P d97f3824f995462e83f56ff3c27bf690 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:02:41.265990 2716 tablet_replica.cc:330] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45: stopping tablet replica
I20260924 03:02:41.266057 2716 raft_consensus.cc:2229] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:02:41.266126 2716 raft_consensus.cc:2258] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:02:41.266224 2714 ts_tablet_manager.cc:1905] T c7ee8f364a044d7abe18ed980f86aab9 P 5e7456a33b6a46d5bf984827a286a2f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:02:41.266439 2716 ts_tablet_manager.cc:1905] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:02:41.266597 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:02:41.266753 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:02:41.267292 2715 ts_tablet_manager.cc:1905] T c7ee8f364a044d7abe18ed980f86aab9 P d97f3824f995462e83f56ff3c27bf690: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:02:41.267887 2714 ts_tablet_manager.cc:1918] T c7ee8f364a044d7abe18ed980f86aab9 P 5e7456a33b6a46d5bf984827a286a2f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:02:41.267963 2714 log.cc:1196] T c7ee8f364a044d7abe18ed980f86aab9 P 5e7456a33b6a46d5bf984827a286a2f7: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-0/wal/wals/c7ee8f364a044d7abe18ed980f86aab9
I20260924 03:02:41.268054 2716 ts_tablet_manager.cc:1918] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:02:41.268118 2716 log.cc:1196] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-1/wal/wals/c7ee8f364a044d7abe18ed980f86aab9
I20260924 03:02:41.268190 2714 ts_tablet_manager.cc:1939] T c7ee8f364a044d7abe18ed980f86aab9 P 5e7456a33b6a46d5bf984827a286a2f7: Deleting consensus metadata
I20260924 03:02:41.268357 2716 ts_tablet_manager.cc:1939] T c7ee8f364a044d7abe18ed980f86aab9 P 6b4bf12bbdea4f47884ccb78a5476f45: Deleting consensus metadata
I20260924 03:02:41.268671 2105 catalog_manager.cc:4841] TS 5e7456a33b6a46d5bf984827a286a2f7 (127.0.98.1:36911): tablet c7ee8f364a044d7abe18ed980f86aab9 (table test-table [id=0fb9fc4545f04deaa474d1d98c98ad1f]) successfully deleted
I20260924 03:02:41.268741 2106 catalog_manager.cc:4841] TS 6b4bf12bbdea4f47884ccb78a5476f45 (127.0.98.2:35617): tablet c7ee8f364a044d7abe18ed980f86aab9 (table test-table [id=0fb9fc4545f04deaa474d1d98c98ad1f]) successfully deleted
I20260924 03:02:41.268996 2715 ts_tablet_manager.cc:1918] T c7ee8f364a044d7abe18ed980f86aab9 P d97f3824f995462e83f56ff3c27bf690: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:02:41.269047 2715 log.cc:1196] T c7ee8f364a044d7abe18ed980f86aab9 P d97f3824f995462e83f56ff3c27bf690: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-2/wal/wals/c7ee8f364a044d7abe18ed980f86aab9
I20260924 03:02:41.269191 2715 ts_tablet_manager.cc:1939] T c7ee8f364a044d7abe18ed980f86aab9 P d97f3824f995462e83f56ff3c27bf690: Deleting consensus metadata
I20260924 03:02:41.269743 2106 catalog_manager.cc:4841] TS d97f3824f995462e83f56ff3c27bf690 (127.0.98.3:39243): tablet c7ee8f364a044d7abe18ed980f86aab9 (table test-table [id=0fb9fc4545f04deaa474d1d98c98ad1f]) successfully deleted
I20260924 03:02:41.270161 2537 tablet_service.cc:1467] Processing CreateTablet for tablet 00dc2c789dcb488bb5e17b4b3e809cce (DEFAULT_TABLE table=test-table [id=de373d0536eb4e3ca933b7f1916fae96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:41.270169 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 00dc2c789dcb488bb5e17b4b3e809cce (DEFAULT_TABLE table=test-table [id=de373d0536eb4e3ca933b7f1916fae96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:41.270283 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00dc2c789dcb488bb5e17b4b3e809cce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:41.270287 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00dc2c789dcb488bb5e17b4b3e809cce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:41.271127 2727 tablet_bootstrap.cc:492] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690: Bootstrap starting.
I20260924 03:02:41.271157 2726 tablet_bootstrap.cc:492] T 00dc2c789dcb488bb5e17b4b3e809cce P 5e7456a33b6a46d5bf984827a286a2f7: Bootstrap starting.
I20260924 03:02:41.271178 2397 tablet_service.cc:1467] Processing CreateTablet for tablet 00dc2c789dcb488bb5e17b4b3e809cce (DEFAULT_TABLE table=test-table [id=de373d0536eb4e3ca933b7f1916fae96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:41.271276 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00dc2c789dcb488bb5e17b4b3e809cce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:41.271452 2727 tablet_bootstrap.cc:654] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:41.271489 2726 tablet_bootstrap.cc:654] T 00dc2c789dcb488bb5e17b4b3e809cce P 5e7456a33b6a46d5bf984827a286a2f7: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:41.272262 2726 tablet_bootstrap.cc:492] T 00dc2c789dcb488bb5e17b4b3e809cce P 5e7456a33b6a46d5bf984827a286a2f7: No bootstrap required, opened a new log
I20260924 03:02:41.272331 2726 ts_tablet_manager.cc:1397] T 00dc2c789dcb488bb5e17b4b3e809cce P 5e7456a33b6a46d5bf984827a286a2f7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 03:02:41.272454 2726 raft_consensus.cc:348] T 00dc2c789dcb488bb5e17b4b3e809cce P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:41.272531 2726 raft_consensus.cc:374] T 00dc2c789dcb488bb5e17b4b3e809cce P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:41.272559 2726 raft_consensus.cc:729] T 00dc2c789dcb488bb5e17b4b3e809cce P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7456a33b6a46d5bf984827a286a2f7, State: Initialized, Role: FOLLOWER
I20260924 03:02:41.272593 2725 tablet_bootstrap.cc:492] T 00dc2c789dcb488bb5e17b4b3e809cce P 6b4bf12bbdea4f47884ccb78a5476f45: Bootstrap starting.
I20260924 03:02:41.272599 2726 consensus_queue.cc:260] T 00dc2c789dcb488bb5e17b4b3e809cce P 5e7456a33b6a46d5bf984827a286a2f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:41.272693 2726 ts_tablet_manager.cc:1428] T 00dc2c789dcb488bb5e17b4b3e809cce P 5e7456a33b6a46d5bf984827a286a2f7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:41.272933 2725 tablet_bootstrap.cc:654] T 00dc2c789dcb488bb5e17b4b3e809cce P 6b4bf12bbdea4f47884ccb78a5476f45: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:41.273072 2727 tablet_bootstrap.cc:492] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690: No bootstrap required, opened a new log
I20260924 03:02:41.273141 2727 ts_tablet_manager.cc:1397] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 03:02:41.273264 2727 raft_consensus.cc:348] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:41.273340 2727 raft_consensus.cc:374] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:41.273368 2727 raft_consensus.cc:729] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d97f3824f995462e83f56ff3c27bf690, State: Initialized, Role: FOLLOWER
I20260924 03:02:41.273414 2727 consensus_queue.cc:260] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:41.273511 2727 ts_tablet_manager.cc:1428] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:41.273561 2725 tablet_bootstrap.cc:492] T 00dc2c789dcb488bb5e17b4b3e809cce P 6b4bf12bbdea4f47884ccb78a5476f45: No bootstrap required, opened a new log
I20260924 03:02:41.273610 2725 ts_tablet_manager.cc:1397] T 00dc2c789dcb488bb5e17b4b3e809cce P 6b4bf12bbdea4f47884ccb78a5476f45: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 03:02:41.273727 2725 raft_consensus.cc:348] T 00dc2c789dcb488bb5e17b4b3e809cce P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:41.273801 2725 raft_consensus.cc:374] T 00dc2c789dcb488bb5e17b4b3e809cce P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:41.273823 2725 raft_consensus.cc:729] T 00dc2c789dcb488bb5e17b4b3e809cce P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b4bf12bbdea4f47884ccb78a5476f45, State: Initialized, Role: FOLLOWER
I20260924 03:02:41.273876 2725 consensus_queue.cc:260] T 00dc2c789dcb488bb5e17b4b3e809cce P 6b4bf12bbdea4f47884ccb78a5476f45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:41.273955 2725 ts_tablet_manager.cc:1428] T 00dc2c789dcb488bb5e17b4b3e809cce P 6b4bf12bbdea4f47884ccb78a5476f45: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:41.437361 2706 raft_consensus.cc:482] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:02:41.437460 2706 raft_consensus.cc:504] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:41.437661 2706 leader_election.cc:288] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e7456a33b6a46d5bf984827a286a2f7 (127.0.98.1:36911), 6b4bf12bbdea4f47884ccb78a5476f45 (127.0.98.2:35617)
I20260924 03:02:41.437939 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00dc2c789dcb488bb5e17b4b3e809cce" candidate_uuid: "d97f3824f995462e83f56ff3c27bf690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" is_pre_election: true
I20260924 03:02:41.437937 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00dc2c789dcb488bb5e17b4b3e809cce" candidate_uuid: "d97f3824f995462e83f56ff3c27bf690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e7456a33b6a46d5bf984827a286a2f7" is_pre_election: true
I20260924 03:02:41.438037 2277 raft_consensus.cc:2454] T 00dc2c789dcb488bb5e17b4b3e809cce P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d97f3824f995462e83f56ff3c27bf690 in term 0.
I20260924 03:02:41.438037 2416 raft_consensus.cc:2454] T 00dc2c789dcb488bb5e17b4b3e809cce P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d97f3824f995462e83f56ff3c27bf690 in term 0.
I20260924 03:02:41.438200 2489 leader_election.cc:302] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e7456a33b6a46d5bf984827a286a2f7, d97f3824f995462e83f56ff3c27bf690; no voters:
I20260924 03:02:41.438298 2706 raft_consensus.cc:2781] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:02:41.438369 2706 raft_consensus.cc:482] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:02:41.438400 2706 raft_consensus.cc:3037] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:41.439133 2706 raft_consensus.cc:504] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:41.439293 2706 leader_election.cc:288] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690 [CANDIDATE]: Term 1 election: Requested vote from peers 5e7456a33b6a46d5bf984827a286a2f7 (127.0.98.1:36911), 6b4bf12bbdea4f47884ccb78a5476f45 (127.0.98.2:35617)
I20260924 03:02:41.439462 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00dc2c789dcb488bb5e17b4b3e809cce" candidate_uuid: "d97f3824f995462e83f56ff3c27bf690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b4bf12bbdea4f47884ccb78a5476f45"
I20260924 03:02:41.439541 2416 raft_consensus.cc:3037] T 00dc2c789dcb488bb5e17b4b3e809cce P 6b4bf12bbdea4f47884ccb78a5476f45 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:41.439461 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00dc2c789dcb488bb5e17b4b3e809cce" candidate_uuid: "d97f3824f995462e83f56ff3c27bf690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e7456a33b6a46d5bf984827a286a2f7"
I20260924 03:02:41.439575 2277 raft_consensus.cc:3037] T 00dc2c789dcb488bb5e17b4b3e809cce P 5e7456a33b6a46d5bf984827a286a2f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:41.440488 2277 raft_consensus.cc:2454] T 00dc2c789dcb488bb5e17b4b3e809cce P 5e7456a33b6a46d5bf984827a286a2f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d97f3824f995462e83f56ff3c27bf690 in term 1.
I20260924 03:02:41.440526 2416 raft_consensus.cc:2454] T 00dc2c789dcb488bb5e17b4b3e809cce P 6b4bf12bbdea4f47884ccb78a5476f45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d97f3824f995462e83f56ff3c27bf690 in term 1.
I20260924 03:02:41.440822 2490 leader_election.cc:302] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b4bf12bbdea4f47884ccb78a5476f45, d97f3824f995462e83f56ff3c27bf690; no voters:
I20260924 03:02:41.440938 2706 raft_consensus.cc:2781] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:41.441097 2706 raft_consensus.cc:686] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690 [term 1 LEADER]: Becoming Leader. State: Replica: d97f3824f995462e83f56ff3c27bf690, State: Running, Role: LEADER
I20260924 03:02:41.441219 2706 consensus_queue.cc:237] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } }
I20260924 03:02:41.441859 2120 catalog_manager.cc:5495] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690 reported cstate change: term changed from 0 to 1, leader changed from <none> to d97f3824f995462e83f56ff3c27bf690 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d97f3824f995462e83f56ff3c27bf690" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d97f3824f995462e83f56ff3c27bf690" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39243 } health_report { overall_health: HEALTHY } } }
I20260924 03:02:41.460208 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:41.457560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.0.98.3:39243)
0924 03:02:41.457739 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:41.457742 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:41.457751 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:02:41.457775 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:41.457777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:41.457789 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:41.457859 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:02:41.458012 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:41.458553 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:41.459270 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:41.459515 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:41.459576 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:41.459818 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:02:41.459852 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:02:41.459912 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:41.459964 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 03:02:41.459990 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":106,"threads_started":1}
I20260924 03:02:41.462618 2416 raft_consensus.cc:1261] T 00dc2c789dcb488bb5e17b4b3e809cce P 6b4bf12bbdea4f47884ccb78a5476f45 [term 1 FOLLOWER]: Refusing update from remote peer d97f3824f995462e83f56ff3c27bf690: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:02:41.462781 2277 raft_consensus.cc:1261] T 00dc2c789dcb488bb5e17b4b3e809cce P 5e7456a33b6a46d5bf984827a286a2f7 [term 1 FOLLOWER]: Refusing update from remote peer d97f3824f995462e83f56ff3c27bf690: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:02:41.462834 2743 consensus_queue.cc:1035] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b4bf12bbdea4f47884ccb78a5476f45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:02:41.462975 2706 consensus_queue.cc:1035] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e7456a33b6a46d5bf984827a286a2f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 03:02:41.467443 2697 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 03:02:42.989319 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:02:42.989428 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:02:42.990263 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a20d6abbc75447a69eb1478b7b10f2f2: Sending DeleteTablet for 3 replicas of tablet 00dc2c789dcb488bb5e17b4b3e809cce
I20260924 03:02:42.990517 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet 00dc2c789dcb488bb5e17b4b3e809cce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:02:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43498
I20260924 03:02:42.990517 2257 tablet_service.cc:1514] Processing DeleteTablet for tablet 00dc2c789dcb488bb5e17b4b3e809cce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:02:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45510
I20260924 03:02:42.990765 2537 tablet_service.cc:1514] Processing DeleteTablet for tablet 00dc2c789dcb488bb5e17b4b3e809cce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:02:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39402
I20260924 03:02:42.990798 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 2192
I20260924 03:02:42.992578 2772 tablet_replica.cc:330] T 00dc2c789dcb488bb5e17b4b3e809cce P 6b4bf12bbdea4f47884ccb78a5476f45: stopping tablet replica
I20260924 03:02:42.992657 2772 raft_consensus.cc:2229] T 00dc2c789dcb488bb5e17b4b3e809cce P 6b4bf12bbdea4f47884ccb78a5476f45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:02:42.992708 2772 raft_consensus.cc:2258] T 00dc2c789dcb488bb5e17b4b3e809cce P 6b4bf12bbdea4f47884ccb78a5476f45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:02:42.993026 2772 ts_tablet_manager.cc:1905] T 00dc2c789dcb488bb5e17b4b3e809cce P 6b4bf12bbdea4f47884ccb78a5476f45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:02:42.994704 2772 ts_tablet_manager.cc:1918] T 00dc2c789dcb488bb5e17b4b3e809cce P 6b4bf12bbdea4f47884ccb78a5476f45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 03:02:42.994767 2772 log.cc:1196] T 00dc2c789dcb488bb5e17b4b3e809cce P 6b4bf12bbdea4f47884ccb78a5476f45: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-1/wal/wals/00dc2c789dcb488bb5e17b4b3e809cce
I20260924 03:02:42.994982 2772 ts_tablet_manager.cc:1939] T 00dc2c789dcb488bb5e17b4b3e809cce P 6b4bf12bbdea4f47884ccb78a5476f45: Deleting consensus metadata
I20260924 03:02:42.995412 2106 catalog_manager.cc:4841] TS 6b4bf12bbdea4f47884ccb78a5476f45 (127.0.98.2:35617): tablet 00dc2c789dcb488bb5e17b4b3e809cce (table test-table [id=de373d0536eb4e3ca933b7f1916fae96]) successfully deleted
I20260924 03:02:42.995741 2774 tablet_replica.cc:330] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690: stopping tablet replica
I20260924 03:02:42.995828 2774 raft_consensus.cc:2229] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:02:42.995898 2774 raft_consensus.cc:2258] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:02:42.996109 2774 ts_tablet_manager.cc:1905] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:02:42.997875 2774 ts_tablet_manager.cc:1918] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
W20260924 03:02:42.997891 2105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36911 (error 108)
I20260924 03:02:42.997939 2774 log.cc:1196] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218948286999-392-0/minicluster-data/ts-2/wal/wals/00dc2c789dcb488bb5e17b4b3e809cce
I20260924 03:02:42.998107 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 2332
I20260924 03:02:42.999536 2774 ts_tablet_manager.cc:1939] T 00dc2c789dcb488bb5e17b4b3e809cce P d97f3824f995462e83f56ff3c27bf690: Deleting consensus metadata
I20260924 03:02:42.999957 2106 catalog_manager.cc:4841] TS d97f3824f995462e83f56ff3c27bf690 (127.0.98.3:39243): tablet 00dc2c789dcb488bb5e17b4b3e809cce (table test-table [id=de373d0536eb4e3ca933b7f1916fae96]) successfully deleted
I20260924 03:02:43.000029 2776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:42.998323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36911 (local address 127.0.0.1:47082)
0924 03:02:42.999916 (+ 1593us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:42.999972 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1563,"thread_start_us":59,"threads_started":1}
W20260924 03:02:43.000133 2105 catalog_manager.cc:4584] TS 5e7456a33b6a46d5bf984827a286a2f7 (127.0.98.1:36911): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 00dc2c789dcb488bb5e17b4b3e809cce: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36911: connect: Connection refused (error 111)
I20260924 03:02:43.004129 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 2472
I20260924 03:02:43.010524 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 2088
2026-09-24T03:02:43Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6389 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:02:43 dist-test-slave-wgsb krb5kdc[2779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:02:43 dist-test-slave-wgsb krb5kdc[2779](info): set up 2 sockets
Sep 24 03:02:43 dist-test-slave-wgsb krb5kdc[2779](info): commencing operation
krb5kdc: starting...
W20260924 03:02:44.059965 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:02:44 dist-test-slave-wgsb krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:02:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:02:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:02:44.203794 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35417
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35417
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:44.290122 2792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:44.290362 2792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:44.290390 2792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:44.291509 2792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:02:44.291553 2792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:44.291568 2792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:44.291584 2792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:02:44.291620 2792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:02:44.293095 2792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:44.293301 2792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:44.293505 2792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:44.299186 2798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:44.299237 2800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:44.299181 2792 server_base.cc:1034] running on GCE node
W20260924 03:02:44.299185 2797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:44.299768 2792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:44.300103 2792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:44.301295 2792 hybrid_clock.cc:648] HybridClock initialized: now 1790218964301277 us; error 49 us; skew 500 ppm
Sep 24 03:02:44 dist-test-slave-wgsb krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:44.311852 2792 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:02:44.312284 2792 webserver.cc:460] Webserver started at http://127.0.98.62:33831/ using document root <none> and password file <none>
I20260924 03:02:44.312551 2792 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:44.312598 2792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:44.312742 2792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:44.313968 2792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "7fd7034e927046a28210e6454d42b719"
format_stamp: "Formatted at 2026-09-24 03:02:44 on dist-test-slave-wgsb"
server_key: "34dc59d005de43fa878afcd2682a27d2"
server_key_iv: "f9fc2fb13baef2d731a76201d228a318"
server_key_version: "encryptionkey@0"
I20260924 03:02:44.314327 2792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "7fd7034e927046a28210e6454d42b719"
format_stamp: "Formatted at 2026-09-24 03:02:44 on dist-test-slave-wgsb"
server_key: "34dc59d005de43fa878afcd2682a27d2"
server_key_iv: "f9fc2fb13baef2d731a76201d228a318"
server_key_version: "encryptionkey@0"
I20260924 03:02:44.315842 2792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:02:44.316658 2807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:44.316907 2792 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:02:44.317003 2792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "7fd7034e927046a28210e6454d42b719"
format_stamp: "Formatted at 2026-09-24 03:02:44 on dist-test-slave-wgsb"
server_key: "34dc59d005de43fa878afcd2682a27d2"
server_key_iv: "f9fc2fb13baef2d731a76201d228a318"
server_key_version: "encryptionkey@0"
I20260924 03:02:44.317075 2792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:44.325300 2792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:44.327780 2792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:44.327939 2792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:44.332171 2792 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35417
I20260924 03:02:44.332304 2859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35417 every 8 connection(s)
I20260924 03:02:44.332780 2792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:02:44.333503 2860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:44.335777 2860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fd7034e927046a28210e6454d42b719: Bootstrap starting.
I20260924 03:02:44.336356 2860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fd7034e927046a28210e6454d42b719: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:44.336648 2860 log.cc:824] T 00000000000000000000000000000000 P 7fd7034e927046a28210e6454d42b719: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:44.337420 2860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fd7034e927046a28210e6454d42b719: No bootstrap required, opened a new log
I20260924 03:02:44.338999 2860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fd7034e927046a28210e6454d42b719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd7034e927046a28210e6454d42b719" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35417 } }
I20260924 03:02:44.339133 2860 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fd7034e927046a28210e6454d42b719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:44.339155 2860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fd7034e927046a28210e6454d42b719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd7034e927046a28210e6454d42b719, State: Initialized, Role: FOLLOWER
I20260924 03:02:44.339325 2860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fd7034e927046a28210e6454d42b719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd7034e927046a28210e6454d42b719" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35417 } }
I20260924 03:02:44.339389 2860 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fd7034e927046a28210e6454d42b719 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:02:44.339411 2860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fd7034e927046a28210e6454d42b719 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:02:44.339463 2860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fd7034e927046a28210e6454d42b719 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:44.339654 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 2792
I20260924 03:02:44.339747 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:02:44.339921 392 external_mini_cluster.cc:1428] Setting key 1ef673fa2ff469d0ada0d6f842000df8
I20260924 03:02:44.340668 2860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fd7034e927046a28210e6454d42b719 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd7034e927046a28210e6454d42b719" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35417 } }
I20260924 03:02:44.340785 2860 leader_election.cc:302] T 00000000000000000000000000000000 P 7fd7034e927046a28210e6454d42b719 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fd7034e927046a28210e6454d42b719; no voters:
I20260924 03:02:44.340968 2860 leader_election.cc:288] T 00000000000000000000000000000000 P 7fd7034e927046a28210e6454d42b719 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:02:44.340996 2865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fd7034e927046a28210e6454d42b719 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:44.341197 2865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fd7034e927046a28210e6454d42b719 [term 1 LEADER]: Becoming Leader. State: Replica: 7fd7034e927046a28210e6454d42b719, State: Running, Role: LEADER
I20260924 03:02:44.341254 2860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fd7034e927046a28210e6454d42b719 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:02:44.341363 2865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fd7034e927046a28210e6454d42b719 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd7034e927046a28210e6454d42b719" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35417 } }
I20260924 03:02:44.341802 2867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fd7034e927046a28210e6454d42b719 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fd7034e927046a28210e6454d42b719. Latest consensus state: current_term: 1 leader_uuid: "7fd7034e927046a28210e6454d42b719" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd7034e927046a28210e6454d42b719" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35417 } } }
I20260924 03:02:44.341910 2867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fd7034e927046a28210e6454d42b719 [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:44.342067 2866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fd7034e927046a28210e6454d42b719 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fd7034e927046a28210e6454d42b719" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd7034e927046a28210e6454d42b719" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35417 } } }
I20260924 03:02:44.342140 2866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fd7034e927046a28210e6454d42b719 [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:44.342096 2874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:02:44.343278 2874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:02:44.344903 2874 catalog_manager.cc:1347] Generated new cluster ID: 87fab281538f4d7d84b20fe990e4f0af
I20260924 03:02:44.344975 2874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:02:44 dist-test-slave-wgsb krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:44.352321 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:44.340488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33088 (local address 127.0.98.62:35417)
0924 03:02:44.340682 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:44.340693 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:44.340725 (+ 32us) server_negotiation.cc:407] Connection header received
0924 03:02:44.340846 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:44.340854 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:44.340910 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:44.341062 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:44.341815 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:44.343217 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:44.344177 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:44.344796 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:44.344903 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:44.348032 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:44.348036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:44.348037 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:44.348049 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:44.350035 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:44.350483 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:44.350487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:44.350488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:44.350641 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:44.350856 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:44.350857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:44.350857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:44.351170 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:44.351333 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:44.351935 (+ 602us) server_negotiation.cc:299] Negotiation successful
0924 03:02:44.352097 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":92,"threads_started":1}
I20260924 03:02:44.358515 2874 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:02:44.359251 2874 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:02:44.369204 2874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fd7034e927046a28210e6454d42b719: Generated new TSK 0
I20260924 03:02:44.369508 2874 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:02:44.408170 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35417
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:44.496913 2886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:44.497138 2886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:44.497164 2886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:44.498332 2886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:44.498368 2886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:44.498394 2886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:02:44.499933 2886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:44.500135 2886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:44.500324 2886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:44.505565 2891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:44.505842 2892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:44.505892 2886 server_base.cc:1034] running on GCE node
W20260924 03:02:44.505892 2894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:44.506289 2886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:44.506541 2886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:44.507684 2886 hybrid_clock.cc:648] HybridClock initialized: now 1790218964507683 us; error 43 us; skew 500 ppm
Sep 24 03:02:44 dist-test-slave-wgsb krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:44.510528 2886 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:02:44.511096 2886 webserver.cc:460] Webserver started at http://127.0.98.1:44839/ using document root <none> and password file <none>
I20260924 03:02:44.511371 2886 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:44.511447 2886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:44.511667 2886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:44.512859 2886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/data/instance:
uuid: "e0ffe6825c4e479d9ddfd4896de4cd84"
format_stamp: "Formatted at 2026-09-24 03:02:44 on dist-test-slave-wgsb"
server_key: "4674f917bac62f3268c9ab3e0eb36bd7"
server_key_iv: "124eb409533c13896e07717a038f6f42"
server_key_version: "encryptionkey@0"
I20260924 03:02:44.513309 2886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e0ffe6825c4e479d9ddfd4896de4cd84"
format_stamp: "Formatted at 2026-09-24 03:02:44 on dist-test-slave-wgsb"
server_key: "4674f917bac62f3268c9ab3e0eb36bd7"
server_key_iv: "124eb409533c13896e07717a038f6f42"
server_key_version: "encryptionkey@0"
I20260924 03:02:44.514727 2886 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:02:44.515552 2901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:44.515813 2886 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:02:44.515919 2886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "e0ffe6825c4e479d9ddfd4896de4cd84"
format_stamp: "Formatted at 2026-09-24 03:02:44 on dist-test-slave-wgsb"
server_key: "4674f917bac62f3268c9ab3e0eb36bd7"
server_key_iv: "124eb409533c13896e07717a038f6f42"
server_key_version: "encryptionkey@0"
I20260924 03:02:44.516013 2886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:44.538295 2886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:44.540428 2886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:44.540594 2886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:44.540845 2886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:44.541224 2886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:44.541297 2886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:44.541339 2886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:44.541366 2886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:44.547772 2886 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39031
I20260924 03:02:44.547814 3014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39031 every 8 connection(s)
I20260924 03:02:44.548154 2886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:02:44.553911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 2886
I20260924 03:02:44.553984 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:02:44.554190 392 external_mini_cluster.cc:1428] Setting key 6c5ed33d90ec051842e38114249941fd
Sep 24 03:02:44 dist-test-slave-wgsb krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:44.559738 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:44.548974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37671 (local address 127.0.98.62:35417)
0924 03:02:44.549081 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:44.549083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:44.549261 (+ 178us) server_negotiation.cc:407] Connection header received
0924 03:02:44.550366 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:44.550368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:44.550383 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:44.550430 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:44.551147 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:44.551782 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:44.552918 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:44.553165 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:44.553260 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:44.556206 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:44.556208 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:44.556208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:44.556218 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:44.557979 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:44.558496 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:44.558497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:44.558498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:44.558551 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:44.558927 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:44.558928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:44.558929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:44.559057 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:44.559117 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:44.559528 (+ 411us) server_negotiation.cc:299] Negotiation successful
0924 03:02:44.559656 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 03:02:44.559996 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:44.548838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35417 (local address 127.0.98.1:37671)
0924 03:02:44.549115 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:44.549166 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:44.550239 (+ 1073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:44.550521 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:44.550525 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:44.550816 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:44.551033 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:44.551037 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:44.551901 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:44.551902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:44.552795 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:44.552797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:44.553270 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:44.553272 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:44.553427 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:44.554127 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:44.554138 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:44.556102 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:44.558070 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:44.558077 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:44.558079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:44.558409 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:44.558621 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:44.558622 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:44.558622 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:44.558809 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:44.559130 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:44.559137 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:44.559329 (+ 192us) client_negotiation.cc:769] Sending connection context
0924 03:02:44.559511 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 03:02:44.559736 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":130,"threads_started":1}
I20260924 03:02:44.560515 3015 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35417
I20260924 03:02:44.560635 3015 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:44.560832 3015 heartbeater.cc:511] Master 127.0.98.62:35417 requested a full tablet report, sending...
I20260924 03:02:44.561429 2824 ts_manager.cc:194] Registered new tserver with Master: e0ffe6825c4e479d9ddfd4896de4cd84 (127.0.98.1:39031)
I20260924 03:02:44.562116 2824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37671
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:02:44.583262 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35417
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:44.668134 3020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:44.668339 3020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:44.668365 3020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:44.669602 3020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:44.669646 3020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:44.669672 3020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:02:44.671185 3020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:44.671423 3020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:44.671692 3020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:44.677070 3026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:44.677074 3025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:44.677074 3028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:44.677572 3020 server_base.cc:1034] running on GCE node
I20260924 03:02:44.677793 3020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:44.678010 3020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:44.679258 3020 hybrid_clock.cc:648] HybridClock initialized: now 1790218964679233 us; error 43 us; skew 500 ppm
Sep 24 03:02:44 dist-test-slave-wgsb krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:44.682238 3020 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:02:44.682675 3020 webserver.cc:460] Webserver started at http://127.0.98.2:34515/ using document root <none> and password file <none>
I20260924 03:02:44.682950 3020 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:44.683000 3020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:44.683139 3020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:44.684355 3020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/data/instance:
uuid: "6e4cbe48cbee4eaba85c8f2a574d4a19"
format_stamp: "Formatted at 2026-09-24 03:02:44 on dist-test-slave-wgsb"
server_key: "aa5c4f6589455e4eab41e0f6cf55c9e8"
server_key_iv: "3d734dc60c7dfd873c611356bc870040"
server_key_version: "encryptionkey@0"
I20260924 03:02:44.684728 3020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6e4cbe48cbee4eaba85c8f2a574d4a19"
format_stamp: "Formatted at 2026-09-24 03:02:44 on dist-test-slave-wgsb"
server_key: "aa5c4f6589455e4eab41e0f6cf55c9e8"
server_key_iv: "3d734dc60c7dfd873c611356bc870040"
server_key_version: "encryptionkey@0"
I20260924 03:02:44.686735 3020 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 03:02:44.688040 3035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:44.688269 3020 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 03:02:44.688370 3020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "6e4cbe48cbee4eaba85c8f2a574d4a19"
format_stamp: "Formatted at 2026-09-24 03:02:44 on dist-test-slave-wgsb"
server_key: "aa5c4f6589455e4eab41e0f6cf55c9e8"
server_key_iv: "3d734dc60c7dfd873c611356bc870040"
server_key_version: "encryptionkey@0"
I20260924 03:02:44.688462 3020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:44.715148 3020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:44.717684 3020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:44.717859 3020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:44.718127 3020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:44.718464 3020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:44.718520 3020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:44.718554 3020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:44.718586 3020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:44.724613 3020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33297
I20260924 03:02:44.724651 3148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33297 every 8 connection(s)
I20260924 03:02:44.724995 3020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:02:44.729038 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 3020
Sep 24 03:02:44 dist-test-slave-wgsb krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:44.730804 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:02:44.734162 392 external_mini_cluster.cc:1428] Setting key 8076654fa36f7464816bcadce57fe3c2
I20260924 03:02:44.743546 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:44.725795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46159 (local address 127.0.98.62:35417)
0924 03:02:44.725863 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:44.725865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:44.726061 (+ 196us) server_negotiation.cc:407] Connection header received
0924 03:02:44.727147 (+ 1086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:44.727148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:44.727162 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:44.727209 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:44.727962 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:44.728477 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:44.730012 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:44.730227 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:44.730312 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:44.738818 (+ 8506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:44.738829 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:44.738830 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:44.738852 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:44.741834 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:44.742458 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:44.742461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:44.742461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:44.742532 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:44.742822 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:44.742822 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:44.742823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:44.742954 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:44.743029 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:44.743344 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 03:02:44.743444 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 03:02:44.743848 3151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:44.725777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35417 (local address 127.0.98.2:46159)
0924 03:02:44.725939 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:44.725975 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:44.727014 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:44.727297 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:44.727301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:44.727568 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:44.727857 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:44.727862 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:44.728580 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:44.728581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:44.729789 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:44.729791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:44.730405 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:44.730406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:44.730527 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:44.731164 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:44.731175 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:44.733398 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:44.741952 (+ 8554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:44.741968 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:44.741971 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:44.742357 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:44.742611 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:44.742612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:44.742612 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:44.742748 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:44.743032 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:44.743040 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:44.743204 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 03:02:44.743323 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 03:02:44.743505 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
I20260924 03:02:44.744341 3149 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35417
I20260924 03:02:44.744441 3149 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:44.744664 3149 heartbeater.cc:511] Master 127.0.98.62:35417 requested a full tablet report, sending...
I20260924 03:02:44.745092 2824 ts_manager.cc:194] Registered new tserver with Master: 6e4cbe48cbee4eaba85c8f2a574d4a19 (127.0.98.2:33297)
I20260924 03:02:44.745494 2824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46159
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:02:44.768565 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35417
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:44.852789 3154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:44.853052 3154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:44.853094 3154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:44.854267 3154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:44.854334 3154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:44.854388 3154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:02:44.856000 3154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:44.856281 3154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:44.856523 3154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:44.861742 3159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:44.861723 3162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:44.861821 3160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:44.861897 3154 server_base.cc:1034] running on GCE node
I20260924 03:02:44.862244 3154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:44.862496 3154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:44.863649 3154 hybrid_clock.cc:648] HybridClock initialized: now 1790218964863643 us; error 35 us; skew 500 ppm
Sep 24 03:02:44 dist-test-slave-wgsb krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:44.866258 3154 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:02:44.866737 3154 webserver.cc:460] Webserver started at http://127.0.98.3:46771/ using document root <none> and password file <none>
I20260924 03:02:44.866986 3154 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:44.867057 3154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:44.867206 3154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:44.868333 3154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/data/instance:
uuid: "cb328ea1e0444f10bdbb8f3c4dad7775"
format_stamp: "Formatted at 2026-09-24 03:02:44 on dist-test-slave-wgsb"
server_key: "5107842d8d70d600b4e083a4e6ce2714"
server_key_iv: "db35dcfb4ed7f9c553dbb894165a0505"
server_key_version: "encryptionkey@0"
I20260924 03:02:44.868693 3154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cb328ea1e0444f10bdbb8f3c4dad7775"
format_stamp: "Formatted at 2026-09-24 03:02:44 on dist-test-slave-wgsb"
server_key: "5107842d8d70d600b4e083a4e6ce2714"
server_key_iv: "db35dcfb4ed7f9c553dbb894165a0505"
server_key_version: "encryptionkey@0"
I20260924 03:02:44.870112 3154 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:02:44.870983 3169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:44.871209 3154 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:44.871381 3154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "cb328ea1e0444f10bdbb8f3c4dad7775"
format_stamp: "Formatted at 2026-09-24 03:02:44 on dist-test-slave-wgsb"
server_key: "5107842d8d70d600b4e083a4e6ce2714"
server_key_iv: "db35dcfb4ed7f9c553dbb894165a0505"
server_key_version: "encryptionkey@0"
I20260924 03:02:44.871469 3154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:44.882838 3154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:44.885839 3154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:44.886029 3154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:44.886281 3154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:44.886657 3154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:44.886714 3154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:44.886758 3154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:44.886786 3154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:44.892609 3154 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39561
I20260924 03:02:44.892680 3282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39561 every 8 connection(s)
I20260924 03:02:44.893008 3154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:02:44.894847 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 3154
I20260924 03:02:44.894940 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:02:44.895099 392 external_mini_cluster.cc:1428] Setting key 7b2dae07a75afc2a9ecaa98ecce40d3e
Sep 24 03:02:44 dist-test-slave-wgsb krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:44.903424 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:44.893712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44199 (local address 127.0.98.62:35417)
0924 03:02:44.893776 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:44.893778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:44.894001 (+ 223us) server_negotiation.cc:407] Connection header received
0924 03:02:44.895040 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:44.895041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:44.895054 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:44.895109 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:44.896057 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:44.896590 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:44.897643 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:44.897818 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:44.897876 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:44.900328 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:44.900329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:44.900329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:44.900336 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:44.901852 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:44.902485 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:44.902487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:44.902487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:44.902548 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:44.902826 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:44.902827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:44.902828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:44.902950 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:44.903013 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:44.903280 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 03:02:44.903350 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 03:02:44.903637 3285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:44.893680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35417 (local address 127.0.98.3:44199)
0924 03:02:44.893883 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:44.893917 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:44.894892 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:44.895201 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:44.895204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:44.895633 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:44.895929 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:44.895936 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:44.896692 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:44.896692 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:44.897569 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:44.897571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:44.897946 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:44.897947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:44.898082 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:44.898753 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:44.898772 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:44.900233 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:44.901974 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:44.901981 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:44.901983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:44.902236 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:44.902629 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:44.902629 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:44.902630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:44.902741 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:44.903016 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:44.903021 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:44.903147 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 03:02:44.903234 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 03:02:44.903375 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":96,"threads_started":1}
I20260924 03:02:44.904186 3283 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35417
I20260924 03:02:44.904294 3283 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:44.904472 3283 heartbeater.cc:511] Master 127.0.98.62:35417 requested a full tablet report, sending...
I20260924 03:02:44.904922 2824 ts_manager.cc:194] Registered new tserver with Master: cb328ea1e0444f10bdbb8f3c4dad7775 (127.0.98.3:39561)
I20260924 03:02:44.905330 2824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44199
I20260924 03:02:44.907179 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 03:02:44.922345 2864 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:02:44.921027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33100 (local address 127.0.98.62:35417)
0924 03:02:44.921159 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:44.921162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:44.921185 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:02:44.921276 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:44.921277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:44.921291 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:44.921353 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:44.922229 (+ 876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33100: BlockingRecv error: recv got EOF from 127.0.0.1:33100 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 03:02:44.923377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 2886
I20260924 03:02:44.929345 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 3020
I20260924 03:02:44.940451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 3154
I20260924 03:02:44.947418 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 2792
2026-09-24T03:02:44Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1941 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:02:45 dist-test-slave-wgsb krb5kdc[3295](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:02:45 dist-test-slave-wgsb krb5kdc[3295](info): set up 2 sockets
Sep 24 03:02:45 dist-test-slave-wgsb krb5kdc[3295](info): commencing operation
krb5kdc: starting...
W20260924 03:02:46.087411 392 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:02:46 dist-test-slave-wgsb krb5kdc[3295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:02:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:02:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:02:46.216877 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40007
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:46.298760 3308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:46.298960 3308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:46.298982 3308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:46.300197 3308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:02:46.300238 3308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:46.300252 3308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:46.300266 3308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:02:46.300318 3308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:02:46.301796 3308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:46.302047 3308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:46.302273 3308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:46.307114 3316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:46.307130 3314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:46.307106 3313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:46.307655 3308 server_base.cc:1034] running on GCE node
I20260924 03:02:46.307881 3308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:46.308157 3308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:46.309297 3308 hybrid_clock.cc:648] HybridClock initialized: now 1790218966309279 us; error 33 us; skew 500 ppm
Sep 24 03:02:46 dist-test-slave-wgsb krb5kdc[3295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:46.312105 3308 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:02:46.312629 3308 webserver.cc:460] Webserver started at http://127.0.98.62:37149/ using document root <none> and password file <none>
I20260924 03:02:46.312876 3308 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:46.312947 3308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:46.313099 3308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:46.314213 3308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "fd21220c226a4536b4aa409ce52987d2"
format_stamp: "Formatted at 2026-09-24 03:02:46 on dist-test-slave-wgsb"
server_key: "38d95f2ae06fbb19886dc6fc92a83250"
server_key_iv: "b2432765f61cdea6feddde773d247f7f"
server_key_version: "encryptionkey@0"
I20260924 03:02:46.314626 3308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "fd21220c226a4536b4aa409ce52987d2"
format_stamp: "Formatted at 2026-09-24 03:02:46 on dist-test-slave-wgsb"
server_key: "38d95f2ae06fbb19886dc6fc92a83250"
server_key_iv: "b2432765f61cdea6feddde773d247f7f"
server_key_version: "encryptionkey@0"
I20260924 03:02:46.316036 3308 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:02:46.316859 3323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:46.317032 3308 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:46.317137 3308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "fd21220c226a4536b4aa409ce52987d2"
format_stamp: "Formatted at 2026-09-24 03:02:46 on dist-test-slave-wgsb"
server_key: "38d95f2ae06fbb19886dc6fc92a83250"
server_key_iv: "b2432765f61cdea6feddde773d247f7f"
server_key_version: "encryptionkey@0"
I20260924 03:02:46.317198 3308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:46.326177 3308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:46.328444 3308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:46.328613 3308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:46.332933 3308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40007
I20260924 03:02:46.332993 3375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40007 every 8 connection(s)
I20260924 03:02:46.333366 3308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:02:46.334331 3376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:46.337149 3376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd21220c226a4536b4aa409ce52987d2: Bootstrap starting.
I20260924 03:02:46.337814 3376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd21220c226a4536b4aa409ce52987d2: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:46.338151 3376 log.cc:824] T 00000000000000000000000000000000 P fd21220c226a4536b4aa409ce52987d2: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:46.338913 3376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd21220c226a4536b4aa409ce52987d2: No bootstrap required, opened a new log
I20260924 03:02:46.340818 3376 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd21220c226a4536b4aa409ce52987d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd21220c226a4536b4aa409ce52987d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40007 } }
I20260924 03:02:46.340969 3376 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd21220c226a4536b4aa409ce52987d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:46.340996 3376 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd21220c226a4536b4aa409ce52987d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd21220c226a4536b4aa409ce52987d2, State: Initialized, Role: FOLLOWER
I20260924 03:02:46.341163 3376 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd21220c226a4536b4aa409ce52987d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd21220c226a4536b4aa409ce52987d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40007 } }
I20260924 03:02:46.341241 3376 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd21220c226a4536b4aa409ce52987d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:02:46.341288 3376 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd21220c226a4536b4aa409ce52987d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:02:46.341349 3376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd21220c226a4536b4aa409ce52987d2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:46.342263 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 3308
I20260924 03:02:46.342366 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:02:46.342368 3376 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd21220c226a4536b4aa409ce52987d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd21220c226a4536b4aa409ce52987d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40007 } }
I20260924 03:02:46.342474 3376 leader_election.cc:302] T 00000000000000000000000000000000 P fd21220c226a4536b4aa409ce52987d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd21220c226a4536b4aa409ce52987d2; no voters:
I20260924 03:02:46.342550 392 external_mini_cluster.cc:1428] Setting key 12f37500ca459133a247ecd6b882187a
I20260924 03:02:46.342664 3376 leader_election.cc:288] T 00000000000000000000000000000000 P fd21220c226a4536b4aa409ce52987d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:02:46.342761 3379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd21220c226a4536b4aa409ce52987d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:46.342943 3376 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd21220c226a4536b4aa409ce52987d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:02:46.343062 3379 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd21220c226a4536b4aa409ce52987d2 [term 1 LEADER]: Becoming Leader. State: Replica: fd21220c226a4536b4aa409ce52987d2, State: Running, Role: LEADER
I20260924 03:02:46.343225 3379 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd21220c226a4536b4aa409ce52987d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd21220c226a4536b4aa409ce52987d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40007 } }
I20260924 03:02:46.343755 3380 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd21220c226a4536b4aa409ce52987d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd21220c226a4536b4aa409ce52987d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd21220c226a4536b4aa409ce52987d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40007 } } }
I20260924 03:02:46.343878 3380 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd21220c226a4536b4aa409ce52987d2 [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:46.344136 3381 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd21220c226a4536b4aa409ce52987d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd21220c226a4536b4aa409ce52987d2. Latest consensus state: current_term: 1 leader_uuid: "fd21220c226a4536b4aa409ce52987d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd21220c226a4536b4aa409ce52987d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40007 } } }
I20260924 03:02:46.344300 3381 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd21220c226a4536b4aa409ce52987d2 [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:46.344466 3390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:02:46.345170 3390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:02:46.346951 3390 catalog_manager.cc:1347] Generated new cluster ID: 879ceffbfeb04fda910c4c5e65aa378c
I20260924 03:02:46.347026 3390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:02:46 dist-test-slave-wgsb krb5kdc[3295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:46.352445 3390 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:02:46.353322 3390 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:02:46.356449 3383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:46.343119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38394 (local address 127.0.98.62:40007)
0924 03:02:46.343385 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:46.343396 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:46.343755 (+ 359us) server_negotiation.cc:407] Connection header received
0924 03:02:46.344032 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:46.344041 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:46.344119 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:46.344275 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:46.345214 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:46.346537 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:46.347713 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:46.348373 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:46.348483 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:46.351659 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:46.351664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:46.351664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:46.351677 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:46.354683 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:46.355135 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:46.355138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:46.355138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:46.355246 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:46.355481 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:46.355482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:46.355483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:46.355798 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:46.355964 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:46.356079 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 03:02:46.356229 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260924 03:02:46.357450 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:02:46.361141 3390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd21220c226a4536b4aa409ce52987d2: Generated new TSK 0
I20260924 03:02:46.361320 3390 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:02:46.373458 3383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:46.369196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38406 (local address 127.0.98.62:40007)
0924 03:02:46.369272 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:46.369275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:46.369462 (+ 187us) server_negotiation.cc:407] Connection header received
0924 03:02:46.369493 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:46.369493 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:46.369505 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:46.369565 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:46.370025 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:46.370600 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:46.371828 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:46.372041 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:46.372100 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:46.372895 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:46.372897 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:46.372897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:02:46.373079 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:46.373097 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:02:46.373102 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:46.373152 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:46.373252 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 03:02:46.373275 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 03:02:46.376246 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 3308
2026-09-24T03:02:46Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1420 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:02:46 dist-test-slave-wgsb krb5kdc[3409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:02:46 dist-test-slave-wgsb krb5kdc[3409](info): set up 2 sockets
Sep 24 03:02:46 dist-test-slave-wgsb krb5kdc[3409](info): commencing operation
krb5kdc: starting...
W20260924 03:02:47.435788 392 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:02:47 dist-test-slave-wgsb krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:02:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:02:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:02:47.563174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37239
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:47.645021 3422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:47.645270 3422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:47.645305 3422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:47.646446 3422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:02:47.646525 3422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:47.646554 3422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:47.646587 3422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:02:47.646611 3422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:02:47.648358 3422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:47.648671 3422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:47.648919 3422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:47.653985 3427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:47.653988 3428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:47.653990 3430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:47.654120 3422 server_base.cc:1034] running on GCE node
I20260924 03:02:47.654484 3422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:47.654778 3422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:47.655952 3422 hybrid_clock.cc:648] HybridClock initialized: now 1790218967655922 us; error 42 us; skew 500 ppm
Sep 24 03:02:47 dist-test-slave-wgsb krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:47.658507 3422 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:02:47.658927 3422 webserver.cc:460] Webserver started at http://127.0.98.62:36081/ using document root <none> and password file <none>
I20260924 03:02:47.659165 3422 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:47.659232 3422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:47.659389 3422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:47.660512 3422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "68758d002427424287324509f2d24dc6"
format_stamp: "Formatted at 2026-09-24 03:02:47 on dist-test-slave-wgsb"
server_key: "d756a06c2eae5840bf0de791139bef80"
server_key_iv: "e9df48388f22a7fc7bf34cced9d22ebf"
server_key_version: "encryptionkey@0"
I20260924 03:02:47.660887 3422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "68758d002427424287324509f2d24dc6"
format_stamp: "Formatted at 2026-09-24 03:02:47 on dist-test-slave-wgsb"
server_key: "d756a06c2eae5840bf0de791139bef80"
server_key_iv: "e9df48388f22a7fc7bf34cced9d22ebf"
server_key_version: "encryptionkey@0"
I20260924 03:02:47.662225 3422 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:02:47.663014 3437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:47.663210 3422 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:47.663302 3422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "68758d002427424287324509f2d24dc6"
format_stamp: "Formatted at 2026-09-24 03:02:47 on dist-test-slave-wgsb"
server_key: "d756a06c2eae5840bf0de791139bef80"
server_key_iv: "e9df48388f22a7fc7bf34cced9d22ebf"
server_key_version: "encryptionkey@0"
I20260924 03:02:47.663376 3422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:47.694660 3422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:47.696831 3422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:47.697010 3422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:47.700958 3422 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37239
I20260924 03:02:47.701002 3489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37239 every 8 connection(s)
I20260924 03:02:47.701376 3422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:02:47.702487 3490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:47.705013 3490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68758d002427424287324509f2d24dc6: Bootstrap starting.
I20260924 03:02:47.705703 3490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68758d002427424287324509f2d24dc6: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:47.706024 3490 log.cc:824] T 00000000000000000000000000000000 P 68758d002427424287324509f2d24dc6: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:47.706776 3490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68758d002427424287324509f2d24dc6: No bootstrap required, opened a new log
I20260924 03:02:47.708906 3490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68758d002427424287324509f2d24dc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68758d002427424287324509f2d24dc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } }
I20260924 03:02:47.709067 3490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68758d002427424287324509f2d24dc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:47.709095 3490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68758d002427424287324509f2d24dc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68758d002427424287324509f2d24dc6, State: Initialized, Role: FOLLOWER
I20260924 03:02:47.709292 3490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68758d002427424287324509f2d24dc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68758d002427424287324509f2d24dc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } }
I20260924 03:02:47.709372 3490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68758d002427424287324509f2d24dc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:02:47.709398 3490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68758d002427424287324509f2d24dc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:02:47.709455 3490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68758d002427424287324509f2d24dc6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:47.709918 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 3422
I20260924 03:02:47.709997 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:02:47.710175 392 external_mini_cluster.cc:1428] Setting key fd7c8a460484726a9527cdbb39b1c5aa
I20260924 03:02:47.710561 3490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68758d002427424287324509f2d24dc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68758d002427424287324509f2d24dc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } }
I20260924 03:02:47.710767 3490 leader_election.cc:302] T 00000000000000000000000000000000 P 68758d002427424287324509f2d24dc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68758d002427424287324509f2d24dc6; no voters:
I20260924 03:02:47.710979 3490 leader_election.cc:288] T 00000000000000000000000000000000 P 68758d002427424287324509f2d24dc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:02:47.711006 3495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68758d002427424287324509f2d24dc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:47.711124 3495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68758d002427424287324509f2d24dc6 [term 1 LEADER]: Becoming Leader. State: Replica: 68758d002427424287324509f2d24dc6, State: Running, Role: LEADER
I20260924 03:02:47.711277 3490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68758d002427424287324509f2d24dc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:02:47.711262 3495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68758d002427424287324509f2d24dc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68758d002427424287324509f2d24dc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } }
I20260924 03:02:47.711882 3497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68758d002427424287324509f2d24dc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68758d002427424287324509f2d24dc6. Latest consensus state: current_term: 1 leader_uuid: "68758d002427424287324509f2d24dc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68758d002427424287324509f2d24dc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } } }
I20260924 03:02:47.711872 3496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68758d002427424287324509f2d24dc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68758d002427424287324509f2d24dc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68758d002427424287324509f2d24dc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } } }
I20260924 03:02:47.712044 3497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68758d002427424287324509f2d24dc6 [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:47.712047 3496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68758d002427424287324509f2d24dc6 [sys.catalog]: This master's current role is: LEADER
W20260924 03:02:47.713681 3510 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 68758d002427424287324509f2d24dc6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:02:47.713732 3510 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:02:47.713995 3503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:02:47.714186 3503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:02:47.715989 3503 catalog_manager.cc:1347] Generated new cluster ID: 0ebe050674fb4cf4890d5cb7f7114c7d
I20260924 03:02:47.716042 3503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:02:47 dist-test-slave-wgsb krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:47.722589 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:47.710741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59344 (local address 127.0.98.62:37239)
0924 03:02:47.710951 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:47.710985 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:47.711019 (+ 34us) server_negotiation.cc:407] Connection header received
0924 03:02:47.711164 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:47.711172 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:47.711228 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:47.711391 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:47.712343 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:47.713817 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:47.714754 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:47.715368 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:47.715490 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:47.718492 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:47.718497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:47.718498 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:47.718511 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:47.720794 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:47.721211 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:47.721214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:47.721214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:47.721358 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:47.721568 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:47.721569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:47.721569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:47.721916 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:47.722107 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:47.722183 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 03:02:47.722351 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":100,"threads_started":1}
I20260924 03:02:47.729176 3503 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:02:47.729725 3503 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:02:47.736847 3503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68758d002427424287324509f2d24dc6: Generated new TSK 0
I20260924 03:02:47.737063 3503 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:02:47.772387 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37239
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:47.854926 3516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:47.855147 3516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:47.855173 3516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:47.856652 3516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:47.856702 3516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:47.856734 3516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:02:47.858275 3516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:47.858551 3516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:47.858817 3516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:47.864009 3522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:47.864110 3524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:47.864035 3521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:47.864290 3516 server_base.cc:1034] running on GCE node
I20260924 03:02:47.864626 3516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:47.864885 3516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:47.866034 3516 hybrid_clock.cc:648] HybridClock initialized: now 1790218967866011 us; error 40 us; skew 500 ppm
Sep 24 03:02:47 dist-test-slave-wgsb krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:47.868870 3516 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:02:47.869347 3516 webserver.cc:460] Webserver started at http://127.0.98.1:45391/ using document root <none> and password file <none>
I20260924 03:02:47.869596 3516 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:47.869644 3516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:47.869776 3516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:47.870826 3516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/data/instance:
uuid: "b9e4fdfa730c4ea5b01600cdf4828b1d"
format_stamp: "Formatted at 2026-09-24 03:02:47 on dist-test-slave-wgsb"
server_key: "d21f43b9bf86cfd2a13645a7b32a2bbc"
server_key_iv: "e12b46024c3eb8c1677f59608a45fca7"
server_key_version: "encryptionkey@0"
I20260924 03:02:47.871179 3516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b9e4fdfa730c4ea5b01600cdf4828b1d"
format_stamp: "Formatted at 2026-09-24 03:02:47 on dist-test-slave-wgsb"
server_key: "d21f43b9bf86cfd2a13645a7b32a2bbc"
server_key_iv: "e12b46024c3eb8c1677f59608a45fca7"
server_key_version: "encryptionkey@0"
I20260924 03:02:47.872717 3516 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:02:47.873489 3531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:47.873670 3516 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:47.873766 3516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "b9e4fdfa730c4ea5b01600cdf4828b1d"
format_stamp: "Formatted at 2026-09-24 03:02:47 on dist-test-slave-wgsb"
server_key: "d21f43b9bf86cfd2a13645a7b32a2bbc"
server_key_iv: "e12b46024c3eb8c1677f59608a45fca7"
server_key_version: "encryptionkey@0"
I20260924 03:02:47.873850 3516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:47.882316 3516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:47.884971 3516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:47.885187 3516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:47.885462 3516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:47.885819 3516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:47.885885 3516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:47.885938 3516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:47.885969 3516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:47.892956 3516 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42193
I20260924 03:02:47.892989 3644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42193 every 8 connection(s)
I20260924 03:02:47.893396 3516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:02:47.897311 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 3516
I20260924 03:02:47.897409 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:02:47.897588 392 external_mini_cluster.cc:1428] Setting key f835699395ace5f88b1c6f8d99000196
Sep 24 03:02:47 dist-test-slave-wgsb krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:47.903367 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:47.894249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54771 (local address 127.0.98.62:37239)
0924 03:02:47.894352 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:47.894354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:47.894514 (+ 160us) server_negotiation.cc:407] Connection header received
0924 03:02:47.895042 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:47.895043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:47.895059 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:47.895117 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:47.896149 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:47.896658 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:47.897974 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:47.898262 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:47.898345 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:47.900526 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:47.900528 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:47.900528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:47.900535 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:47.902017 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:47.902417 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:47.902418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:47.902418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:47.902478 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:47.902730 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:47.902731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:47.902732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:47.902848 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:47.902923 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:47.903191 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 03:02:47.903275 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 03:02:47.903578 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:47.894168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.1:54771)
0924 03:02:47.894381 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:47.894426 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:47.894936 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:47.895191 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:47.895194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:47.895857 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:47.896052 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:47.896057 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:47.896752 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:47.896753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:47.897891 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:47.897894 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:47.898354 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:47.898356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:47.898499 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:47.899048 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:47.899056 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:47.900393 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:47.902103 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:47.902109 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:47.902112 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:47.902332 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:47.902548 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:47.902549 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:47.902549 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:47.902653 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:47.902914 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:47.902919 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:47.903060 (+ 141us) client_negotiation.cc:769] Sending connection context
0924 03:02:47.903170 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 03:02:47.903324 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":92,"threads_started":1}
I20260924 03:02:47.904361 3645 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37239
I20260924 03:02:47.904474 3645 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:47.904647 3645 heartbeater.cc:511] Master 127.0.98.62:37239 requested a full tablet report, sending...
I20260924 03:02:47.905237 3454 ts_manager.cc:194] Registered new tserver with Master: b9e4fdfa730c4ea5b01600cdf4828b1d (127.0.98.1:42193)
I20260924 03:02:47.906281 3454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54771
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:02:47.926222 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37239
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:48.017251 3650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:48.017467 3650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:48.017494 3650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:48.019037 3650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:48.019089 3650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:48.019119 3650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:02:48.020686 3650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:48.020888 3650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:48.021150 3650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:48.026381 3658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:48.026459 3656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:48.026391 3655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:48.026669 3650 server_base.cc:1034] running on GCE node
I20260924 03:02:48.026897 3650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:48.027158 3650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:48.028362 3650 hybrid_clock.cc:648] HybridClock initialized: now 1790218968028338 us; error 46 us; skew 500 ppm
Sep 24 03:02:48 dist-test-slave-wgsb krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:48.030998 3650 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:02:48.031512 3650 webserver.cc:460] Webserver started at http://127.0.98.2:35455/ using document root <none> and password file <none>
I20260924 03:02:48.031767 3650 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:48.031836 3650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:48.031996 3650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:48.033129 3650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/data/instance:
uuid: "b92bc77c8653409aaac614064ca184ef"
format_stamp: "Formatted at 2026-09-24 03:02:48 on dist-test-slave-wgsb"
server_key: "2fc7aea8876332c93448fd5b14c01817"
server_key_iv: "31f4b457df7e0b61cf5474f7d296a498"
server_key_version: "encryptionkey@0"
I20260924 03:02:48.033515 3650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b92bc77c8653409aaac614064ca184ef"
format_stamp: "Formatted at 2026-09-24 03:02:48 on dist-test-slave-wgsb"
server_key: "2fc7aea8876332c93448fd5b14c01817"
server_key_iv: "31f4b457df7e0b61cf5474f7d296a498"
server_key_version: "encryptionkey@0"
I20260924 03:02:48.034863 3650 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:02:48.035679 3665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:48.035856 3650 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:48.035955 3650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "b92bc77c8653409aaac614064ca184ef"
format_stamp: "Formatted at 2026-09-24 03:02:48 on dist-test-slave-wgsb"
server_key: "2fc7aea8876332c93448fd5b14c01817"
server_key_iv: "31f4b457df7e0b61cf5474f7d296a498"
server_key_version: "encryptionkey@0"
I20260924 03:02:48.036034 3650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:48.044238 3650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:48.046787 3650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:48.046974 3650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:48.047250 3650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:48.047679 3650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:48.047737 3650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:48.047786 3650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:48.047842 3650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:48.054695 3650 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36329
I20260924 03:02:48.054737 3778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36329 every 8 connection(s)
I20260924 03:02:48.055070 3650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 03:02:48 dist-test-slave-wgsb krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:48.061690 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 3650
I20260924 03:02:48.061805 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:02:48.062013 392 external_mini_cluster.cc:1428] Setting key 05ed8482ad4918e31e62d7713eea323d
I20260924 03:02:48.065150 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:48.055848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53289 (local address 127.0.98.62:37239)
0924 03:02:48.055937 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:48.055939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:48.056123 (+ 184us) server_negotiation.cc:407] Connection header received
0924 03:02:48.056620 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:48.056621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:48.056632 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:48.056698 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:48.057612 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:48.058107 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:48.059140 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:48.059372 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:48.059643 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:48.061766 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:48.061768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:48.061769 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:48.061779 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:48.063642 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:48.064048 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:48.064049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:48.064050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:48.064112 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:48.064436 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:48.064436 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:48.064437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:48.064552 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:48.064609 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:48.065006 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 03:02:48.065076 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 03:02:48.065433 3781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:48.055806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.2:53289)
0924 03:02:48.055996 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:48.056035 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:48.056514 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:48.056753 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:48.056757 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:48.057319 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:48.057520 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:48.057525 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:48.058183 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:48.058183 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:48.059030 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:48.059033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:48.059458 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:48.059458 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:48.059582 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:48.060133 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:48.060142 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:48.061630 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:48.063733 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:48.063739 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:48.063741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:48.063972 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:48.064195 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:48.064196 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:48.064196 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:48.064363 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:48.064623 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:48.064630 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:48.064806 (+ 176us) client_negotiation.cc:769] Sending connection context
0924 03:02:48.064968 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 03:02:48.065175 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":107,"threads_started":1}
I20260924 03:02:48.065973 3779 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37239
I20260924 03:02:48.066056 3779 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:48.066244 3779 heartbeater.cc:511] Master 127.0.98.62:37239 requested a full tablet report, sending...
I20260924 03:02:48.066666 3454 ts_manager.cc:194] Registered new tserver with Master: b92bc77c8653409aaac614064ca184ef (127.0.98.2:36329)
I20260924 03:02:48.067227 3454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53289
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:02:48.090219 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37239
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:48.173331 3784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:48.173542 3784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:48.173564 3784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:48.174675 3784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:48.174712 3784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:48.174738 3784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:02:48.176204 3784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:48.176393 3784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:48.176565 3784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:48.181794 3792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:48.181784 3790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:48.181806 3789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:48.181824 3784 server_base.cc:1034] running on GCE node
I20260924 03:02:48.182204 3784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:48.182525 3784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:48.183693 3784 hybrid_clock.cc:648] HybridClock initialized: now 1790218968183681 us; error 29 us; skew 500 ppm
Sep 24 03:02:48 dist-test-slave-wgsb krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:48.186328 3784 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:02:48.186757 3784 webserver.cc:460] Webserver started at http://127.0.98.3:45229/ using document root <none> and password file <none>
I20260924 03:02:48.186987 3784 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:48.187052 3784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:48.187197 3784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:48.188382 3784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/data/instance:
uuid: "e7b5d8ca9ff2417092b9d0ed07cdc979"
format_stamp: "Formatted at 2026-09-24 03:02:48 on dist-test-slave-wgsb"
server_key: "e8d3af3ee19b8948aa040e3ba4a4acf0"
server_key_iv: "c3bfa535dfe777d3b7b218a05260cb83"
server_key_version: "encryptionkey@0"
I20260924 03:02:48.188735 3784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e7b5d8ca9ff2417092b9d0ed07cdc979"
format_stamp: "Formatted at 2026-09-24 03:02:48 on dist-test-slave-wgsb"
server_key: "e8d3af3ee19b8948aa040e3ba4a4acf0"
server_key_iv: "c3bfa535dfe777d3b7b218a05260cb83"
server_key_version: "encryptionkey@0"
I20260924 03:02:48.190039 3784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:02:48.190794 3799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:48.190971 3784 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:02:48.191064 3784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "e7b5d8ca9ff2417092b9d0ed07cdc979"
format_stamp: "Formatted at 2026-09-24 03:02:48 on dist-test-slave-wgsb"
server_key: "e8d3af3ee19b8948aa040e3ba4a4acf0"
server_key_iv: "c3bfa535dfe777d3b7b218a05260cb83"
server_key_version: "encryptionkey@0"
I20260924 03:02:48.191147 3784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:48.214438 3784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:48.216979 3784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:48.217161 3784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:48.217423 3784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:48.217764 3784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:48.217829 3784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:48.217927 3784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:48.217957 3784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:48.224439 3784 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44713
I20260924 03:02:48.224509 3912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44713 every 8 connection(s)
I20260924 03:02:48.224864 3784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:02:48.225188 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 3784
I20260924 03:02:48.225275 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218948286999-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:02:48.225483 392 external_mini_cluster.cc:1428] Setting key c2f98514cbb1a362802e24118e8e86da
Sep 24 03:02:48 dist-test-slave-wgsb krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:48.235056 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:48.225722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39233 (local address 127.0.98.62:37239)
0924 03:02:48.225769 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:48.225772 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:48.226053 (+ 281us) server_negotiation.cc:407] Connection header received
0924 03:02:48.226636 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:48.226637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:48.226650 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:48.226718 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:48.227677 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:48.228371 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:48.229380 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:48.229584 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:48.229652 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:48.232204 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:48.232205 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:48.232205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:48.232213 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:48.233860 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:48.234222 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:48.234223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:48.234223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:48.234276 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:48.234480 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:48.234480 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:48.234481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:48.234590 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:48.234644 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:48.234919 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 03:02:48.234989 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:02:48.235184 3915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:48.225716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.3:39233)
0924 03:02:48.225891 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:48.225948 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:48.226501 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:48.226772 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:48.226776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:48.227345 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:48.227540 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:48.227544 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:48.228470 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:48.228470 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:48.229285 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:48.229287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:48.229666 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:48.229667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:48.229767 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:48.230310 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:48.230320 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:48.232107 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:48.233943 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:48.233948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:48.233949 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:48.234156 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:48.234333 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:48.234333 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:48.234333 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:48.234424 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:48.234644 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:48.234648 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:48.234764 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 03:02:48.234846 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 03:02:48.234970 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":107,"threads_started":1}
I20260924 03:02:48.235785 3913 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37239
I20260924 03:02:48.235903 3913 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:48.236109 3913 heartbeater.cc:511] Master 127.0.98.62:37239 requested a full tablet report, sending...
I20260924 03:02:48.236732 3454 ts_manager.cc:194] Registered new tserver with Master: e7b5d8ca9ff2417092b9d0ed07cdc979 (127.0.98.3:44713)
I20260924 03:02:48.237265 3454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39233
I20260924 03:02:48.237500 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 03:02:48 dist-test-slave-wgsb krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218968, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 03:02:48 dist-test-slave-wgsb krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218968, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:48.260439 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:48.253115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59348 (local address 127.0.98.62:37239)
0924 03:02:48.253205 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:48.253208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:48.253309 (+ 101us) server_negotiation.cc:407] Connection header received
0924 03:02:48.253410 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:48.253411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:48.253421 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:48.253496 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:48.254107 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:48.254614 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:48.255395 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:48.255586 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:48.255718 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:48.258043 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:48.258050 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:48.258050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:48.258057 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:48.259259 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:48.259629 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:48.259630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:48.259631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:48.259690 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:48.259887 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:48.259888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:48.259889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:48.260044 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:48.260155 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:48.260258 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 03:02:48.260336 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 03:02:48.260767 3454 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59348
I20260924 03:02:48.261168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 3516
I20260924 03:02:48.266173 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 3650
I20260924 03:02:48.271062 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 3784
I20260924 03:02:48.275301 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 3422
2026-09-24T03:02:48Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1900 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:02:48 dist-test-slave-wgsb krb5kdc[3925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:02:48 dist-test-slave-wgsb krb5kdc[3925](info): set up 2 sockets
Sep 24 03:02:48 dist-test-slave-wgsb krb5kdc[3925](info): commencing operation
krb5kdc: starting...
W20260924 03:02:49.315729 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:02:49 dist-test-slave-wgsb krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:02:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:02:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:02:49.442153 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40165
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:49.522271 3938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:49.522509 3938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:49.522543 3938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:49.523794 3938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:02:49.523862 3938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:49.523890 3938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:49.523921 3938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:02:49.523948 3938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:02:49.525405 3938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:49.525693 3938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:49.525913 3938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:49.530879 3946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:49.530879 3943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:49.530912 3944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:49.531252 3938 server_base.cc:1034] running on GCE node
I20260924 03:02:49.531627 3938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:49.531941 3938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:49.533107 3938 hybrid_clock.cc:648] HybridClock initialized: now 1790218969533100 us; error 49 us; skew 500 ppm
Sep 24 03:02:49 dist-test-slave-wgsb krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:49.541528 3938 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:02:49.541985 3938 webserver.cc:460] Webserver started at http://127.0.98.62:36719/ using document root <none> and password file <none>
I20260924 03:02:49.542213 3938 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:49.542259 3938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:49.542428 3938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:49.543526 3938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "8f01e39d0b3a401a8ba7efcc3643b2a8"
format_stamp: "Formatted at 2026-09-24 03:02:49 on dist-test-slave-wgsb"
server_key: "e1da577d076d8571a2b6b31de6821eac"
server_key_iv: "cf3ef06aca37459a6f2a8e7de00f8b88"
server_key_version: "encryptionkey@0"
I20260924 03:02:49.543985 3938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "8f01e39d0b3a401a8ba7efcc3643b2a8"
format_stamp: "Formatted at 2026-09-24 03:02:49 on dist-test-slave-wgsb"
server_key: "e1da577d076d8571a2b6b31de6821eac"
server_key_iv: "cf3ef06aca37459a6f2a8e7de00f8b88"
server_key_version: "encryptionkey@0"
I20260924 03:02:49.545328 3938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:49.546170 3953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:49.546370 3938 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:49.546522 3938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "8f01e39d0b3a401a8ba7efcc3643b2a8"
format_stamp: "Formatted at 2026-09-24 03:02:49 on dist-test-slave-wgsb"
server_key: "e1da577d076d8571a2b6b31de6821eac"
server_key_iv: "cf3ef06aca37459a6f2a8e7de00f8b88"
server_key_version: "encryptionkey@0"
I20260924 03:02:49.546582 3938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:49.564891 3938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:49.567150 3938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:49.567274 3938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:49.571307 3938 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40165
I20260924 03:02:49.571341 4005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40165 every 8 connection(s)
I20260924 03:02:49.571801 3938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:02:49.572759 4006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:49.575199 4006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f01e39d0b3a401a8ba7efcc3643b2a8: Bootstrap starting.
I20260924 03:02:49.575852 4006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f01e39d0b3a401a8ba7efcc3643b2a8: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:49.576182 4006 log.cc:824] T 00000000000000000000000000000000 P 8f01e39d0b3a401a8ba7efcc3643b2a8: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:49.576963 4006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f01e39d0b3a401a8ba7efcc3643b2a8: No bootstrap required, opened a new log
I20260924 03:02:49.577280 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 3938
I20260924 03:02:49.577359 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:02:49.577623 392 external_mini_cluster.cc:1428] Setting key cbf07d572d47af5b889c9937cca83486
I20260924 03:02:49.579294 4006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f01e39d0b3a401a8ba7efcc3643b2a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f01e39d0b3a401a8ba7efcc3643b2a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40165 } }
I20260924 03:02:49.579474 4006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f01e39d0b3a401a8ba7efcc3643b2a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:49.579535 4006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f01e39d0b3a401a8ba7efcc3643b2a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f01e39d0b3a401a8ba7efcc3643b2a8, State: Initialized, Role: FOLLOWER
I20260924 03:02:49.579703 4006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f01e39d0b3a401a8ba7efcc3643b2a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f01e39d0b3a401a8ba7efcc3643b2a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40165 } }
I20260924 03:02:49.579784 4006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f01e39d0b3a401a8ba7efcc3643b2a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:02:49.579807 4006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f01e39d0b3a401a8ba7efcc3643b2a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:02:49.579869 4006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f01e39d0b3a401a8ba7efcc3643b2a8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:49.580931 4006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f01e39d0b3a401a8ba7efcc3643b2a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f01e39d0b3a401a8ba7efcc3643b2a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40165 } }
I20260924 03:02:49.581040 4006 leader_election.cc:302] T 00000000000000000000000000000000 P 8f01e39d0b3a401a8ba7efcc3643b2a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f01e39d0b3a401a8ba7efcc3643b2a8; no voters:
I20260924 03:02:49.581197 4006 leader_election.cc:288] T 00000000000000000000000000000000 P 8f01e39d0b3a401a8ba7efcc3643b2a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:02:49.581290 4011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f01e39d0b3a401a8ba7efcc3643b2a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:49.581454 4006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f01e39d0b3a401a8ba7efcc3643b2a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:02:49.581696 4011 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f01e39d0b3a401a8ba7efcc3643b2a8 [term 1 LEADER]: Becoming Leader. State: Replica: 8f01e39d0b3a401a8ba7efcc3643b2a8, State: Running, Role: LEADER
I20260924 03:02:49.581925 4011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f01e39d0b3a401a8ba7efcc3643b2a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f01e39d0b3a401a8ba7efcc3643b2a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40165 } }
I20260924 03:02:49.582724 4013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f01e39d0b3a401a8ba7efcc3643b2a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f01e39d0b3a401a8ba7efcc3643b2a8. Latest consensus state: current_term: 1 leader_uuid: "8f01e39d0b3a401a8ba7efcc3643b2a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f01e39d0b3a401a8ba7efcc3643b2a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40165 } } }
I20260924 03:02:49.582821 4013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f01e39d0b3a401a8ba7efcc3643b2a8 [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:49.583130 4024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:02:49.583537 4012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f01e39d0b3a401a8ba7efcc3643b2a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f01e39d0b3a401a8ba7efcc3643b2a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f01e39d0b3a401a8ba7efcc3643b2a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40165 } } }
I20260924 03:02:49.583630 4012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f01e39d0b3a401a8ba7efcc3643b2a8 [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:49.583827 4024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 03:02:49 dist-test-slave-wgsb krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:49.586201 4024 catalog_manager.cc:1347] Generated new cluster ID: 4bf4ec13ba9341b3aa5c1b7053db1ba6
I20260924 03:02:49.586241 4024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:02:49.590433 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:49.578147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35214 (local address 127.0.98.62:40165)
0924 03:02:49.578357 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:49.578368 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:49.578400 (+ 32us) server_negotiation.cc:407] Connection header received
0924 03:02:49.579254 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:49.579263 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:49.579338 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:49.579499 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:49.580303 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:49.581723 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:49.583032 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:49.583735 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:49.583857 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:49.586718 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:49.586722 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:49.586722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:49.586735 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:49.588781 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:49.589168 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:49.589182 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:49.589182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:49.589303 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:49.589505 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:49.589506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:49.589506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:49.589814 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:49.589984 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:49.590062 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 03:02:49.590200 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":91,"threads_started":1}
I20260924 03:02:49.611027 4024 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:02:49.611651 4024 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:02:49.614675 4024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f01e39d0b3a401a8ba7efcc3643b2a8: Generated new TSK 0
I20260924 03:02:49.614869 4024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:02:49.642591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40165
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:49.723902 4032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:49.724141 4032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:49.724180 4032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:49.725368 4032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:49.725438 4032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:49.725483 4032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:02:49.726997 4032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:49.727265 4032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:49.727494 4032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:49.732663 4037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:49.732663 4040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:49.732686 4038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:49.733136 4032 server_base.cc:1034] running on GCE node
I20260924 03:02:49.733438 4032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:49.733718 4032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:49.734900 4032 hybrid_clock.cc:648] HybridClock initialized: now 1790218969734886 us; error 42 us; skew 500 ppm
Sep 24 03:02:49 dist-test-slave-wgsb krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:49.737555 4032 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:02:49.738032 4032 webserver.cc:460] Webserver started at http://127.0.98.1:33999/ using document root <none> and password file <none>
I20260924 03:02:49.738260 4032 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:49.738304 4032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:49.738444 4032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:49.739480 4032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-0/data/instance:
uuid: "acbec27677d04a39bf184da7f325b924"
format_stamp: "Formatted at 2026-09-24 03:02:49 on dist-test-slave-wgsb"
server_key: "3d4fe6f82ec0179c3aaf90fa11be7c63"
server_key_iv: "3160ba816ae02e035b0d22ae3a4e3dab"
server_key_version: "encryptionkey@0"
I20260924 03:02:49.739921 4032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-0/wal/instance:
uuid: "acbec27677d04a39bf184da7f325b924"
format_stamp: "Formatted at 2026-09-24 03:02:49 on dist-test-slave-wgsb"
server_key: "3d4fe6f82ec0179c3aaf90fa11be7c63"
server_key_iv: "3160ba816ae02e035b0d22ae3a4e3dab"
server_key_version: "encryptionkey@0"
I20260924 03:02:49.741201 4032 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:49.741994 4047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:49.742185 4032 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:49.742292 4032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "acbec27677d04a39bf184da7f325b924"
format_stamp: "Formatted at 2026-09-24 03:02:49 on dist-test-slave-wgsb"
server_key: "3d4fe6f82ec0179c3aaf90fa11be7c63"
server_key_iv: "3160ba816ae02e035b0d22ae3a4e3dab"
server_key_version: "encryptionkey@0"
I20260924 03:02:49.742373 4032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:49.761490 4032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:49.763918 4032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:49.764081 4032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:49.764331 4032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:49.764655 4032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:49.764710 4032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:49.764745 4032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:49.764772 4032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:49.770710 4032 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46055
I20260924 03:02:49.770753 4160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46055 every 8 connection(s)
I20260924 03:02:49.771065 4032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 03:02:49 dist-test-slave-wgsb krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:49.777912 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 4032
I20260924 03:02:49.778035 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:02:49.778283 392 external_mini_cluster.cc:1428] Setting key 1765ccd204ea3db61085bad03b945649
I20260924 03:02:49.781787 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:49.771800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55247 (local address 127.0.98.62:40165)
0924 03:02:49.771885 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:49.771887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:49.772174 (+ 287us) server_negotiation.cc:407] Connection header received
0924 03:02:49.773142 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:49.773143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:49.773159 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:49.773216 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:49.773849 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:49.774404 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:49.775442 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:49.775736 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:49.775821 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:49.778253 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:49.778254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:49.778254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:49.778262 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:49.780052 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:49.780660 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:49.780662 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:49.780662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:49.780740 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:49.781039 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:49.781040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:49.781040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:49.781190 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:49.781268 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:49.781552 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 03:02:49.781667 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 03:02:49.781962 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:49.771778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40165 (local address 127.0.98.1:55247)
0924 03:02:49.771983 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:49.772081 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:49.773025 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:49.773292 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:49.773296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:49.773557 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:49.773748 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:49.773753 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:49.774504 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:49.774505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:49.775335 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:49.775337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:49.775832 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:49.775833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:49.775961 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:49.776498 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:49.776507 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:49.778124 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:49.780149 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:49.780158 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:49.780161 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:49.780552 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:49.780830 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:49.780831 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:49.780832 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:49.780964 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:49.781286 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:49.781291 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:49.781413 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 03:02:49.781509 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 03:02:49.781703 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":105,"threads_started":1}
I20260924 03:02:49.782550 4161 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40165
I20260924 03:02:49.782671 4161 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:49.782873 4161 heartbeater.cc:511] Master 127.0.98.62:40165 requested a full tablet report, sending...
I20260924 03:02:49.783430 3970 ts_manager.cc:194] Registered new tserver with Master: acbec27677d04a39bf184da7f325b924 (127.0.98.1:46055)
I20260924 03:02:49.784070 3970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55247
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:02:49.806191 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40165
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:49.890966 4166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:49.891184 4166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:49.891211 4166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:49.892460 4166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:49.892493 4166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:49.892518 4166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:02:49.893884 4166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:49.894060 4166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:49.894240 4166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:49.899473 4171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:49.899437 4174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:49.899528 4172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:49.899905 4166 server_base.cc:1034] running on GCE node
I20260924 03:02:49.900094 4166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:49.900291 4166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:49.901489 4166 hybrid_clock.cc:648] HybridClock initialized: now 1790218969901473 us; error 28 us; skew 500 ppm
Sep 24 03:02:49 dist-test-slave-wgsb krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:49.904295 4166 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:02:49.904695 4166 webserver.cc:460] Webserver started at http://127.0.98.2:33741/ using document root <none> and password file <none>
I20260924 03:02:49.904923 4166 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:49.904994 4166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:49.905148 4166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:49.906206 4166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-1/data/instance:
uuid: "e96e2fd39f5b447ea487e5048a611d7e"
format_stamp: "Formatted at 2026-09-24 03:02:49 on dist-test-slave-wgsb"
server_key: "38d64d3a3e1b4dcc02976de9f83dbfb1"
server_key_iv: "07f99643a91c4a9061136a7644501659"
server_key_version: "encryptionkey@0"
I20260924 03:02:49.906574 4166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e96e2fd39f5b447ea487e5048a611d7e"
format_stamp: "Formatted at 2026-09-24 03:02:49 on dist-test-slave-wgsb"
server_key: "38d64d3a3e1b4dcc02976de9f83dbfb1"
server_key_iv: "07f99643a91c4a9061136a7644501659"
server_key_version: "encryptionkey@0"
I20260924 03:02:49.907927 4166 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:02:49.908687 4181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:49.908885 4166 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:02:49.909003 4166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "e96e2fd39f5b447ea487e5048a611d7e"
format_stamp: "Formatted at 2026-09-24 03:02:49 on dist-test-slave-wgsb"
server_key: "38d64d3a3e1b4dcc02976de9f83dbfb1"
server_key_iv: "07f99643a91c4a9061136a7644501659"
server_key_version: "encryptionkey@0"
I20260924 03:02:49.909088 4166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:49.920611 4166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:49.922912 4166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:49.923115 4166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:49.923374 4166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:49.923872 4166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:49.923930 4166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:49.923969 4166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:49.924012 4166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:49.929962 4166 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34367
I20260924 03:02:49.930002 4294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34367 every 8 connection(s)
I20260924 03:02:49.930349 4166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:02:49.931316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 4166
I20260924 03:02:49.931460 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:02:49.931704 392 external_mini_cluster.cc:1428] Setting key 12fc6710143167e628bd47c3d217959b
Sep 24 03:02:49 dist-test-slave-wgsb krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 03:02:49.941530 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:49.931160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47165 (local address 127.0.98.62:40165)
0924 03:02:49.931238 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:49.931241 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:49.931503 (+ 262us) server_negotiation.cc:407] Connection header received
0924 03:02:49.932791 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:49.932792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:49.932804 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:49.932868 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:49.933561 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:49.934155 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:49.935195 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:49.935390 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:49.935463 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:49.938280 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:49.938282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:49.938282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:49.938291 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:49.939960 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:49.940518 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:49.940518 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:49.940519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:49.940608 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:49.940875 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:49.940876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:49.940877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:49.940991 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:49.941069 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:49.941340 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 03:02:49.941424 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 03:02:49.941735 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:49.931075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40165 (local address 127.0.98.2:47165)
0924 03:02:49.931276 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:49.931328 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:49.932689 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:49.932945 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:49.932949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:49.933263 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:49.933470 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:49.933474 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:49.934260 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:49.934262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:49.935091 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:49.935094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:49.935479 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:49.935480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:49.935658 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:49.936274 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:49.936283 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:49.938158 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:49.940056 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:49.940063 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:49.940066 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:49.940375 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:49.940685 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:49.940686 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:49.940687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:49.940796 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:49.941059 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:49.941065 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:49.941209 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 03:02:49.941321 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 03:02:49.941480 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":99,"threads_started":1}
I20260924 03:02:49.942278 4295 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40165
I20260924 03:02:49.942359 4295 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:49.942553 4295 heartbeater.cc:511] Master 127.0.98.62:40165 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 03:02:49.942999 3970 ts_manager.cc:194] Registered new tserver with Master: e96e2fd39f5b447ea487e5048a611d7e (127.0.98.2:34367)
I20260924 03:02:49.943435 3970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:02:49.959092 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40165
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:50.044061 4300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:50.044332 4300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:50.044381 4300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:50.045665 4300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:50.045724 4300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:50.045774 4300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:02:50.047339 4300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:50.047676 4300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:50.047909 4300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:50.053054 4305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:50.053063 4306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:50.053063 4308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:50.053169 4300 server_base.cc:1034] running on GCE node
I20260924 03:02:50.053624 4300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:50.053858 4300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:50.055004 4300 hybrid_clock.cc:648] HybridClock initialized: now 1790218970054979 us; error 41 us; skew 500 ppm
Sep 24 03:02:50 dist-test-slave-wgsb krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:50.057686 4300 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:02:50.058343 4300 webserver.cc:460] Webserver started at http://127.0.98.3:37567/ using document root <none> and password file <none>
I20260924 03:02:50.058647 4300 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:50.058703 4300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:50.058880 4300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:50.060029 4300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-2/data/instance:
uuid: "25f2fc12a6d24997af8384c0c3c39413"
format_stamp: "Formatted at 2026-09-24 03:02:50 on dist-test-slave-wgsb"
server_key: "5623fa68d3a9b50afa6bef17de5c01fb"
server_key_iv: "4197d3edf48045ec419cfb6d648152e3"
server_key_version: "encryptionkey@0"
I20260924 03:02:50.060423 4300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-2/wal/instance:
uuid: "25f2fc12a6d24997af8384c0c3c39413"
format_stamp: "Formatted at 2026-09-24 03:02:50 on dist-test-slave-wgsb"
server_key: "5623fa68d3a9b50afa6bef17de5c01fb"
server_key_iv: "4197d3edf48045ec419cfb6d648152e3"
server_key_version: "encryptionkey@0"
I20260924 03:02:50.061851 4300 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:02:50.062660 4315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:50.062865 4300 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:50.062969 4300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "25f2fc12a6d24997af8384c0c3c39413"
format_stamp: "Formatted at 2026-09-24 03:02:50 on dist-test-slave-wgsb"
server_key: "5623fa68d3a9b50afa6bef17de5c01fb"
server_key_iv: "4197d3edf48045ec419cfb6d648152e3"
server_key_version: "encryptionkey@0"
I20260924 03:02:50.063045 4300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:50.080945 4300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:50.083320 4300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:50.083500 4300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:50.083756 4300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:50.084071 4300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:50.084124 4300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:50.084167 4300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:50.084194 4300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:50.090061 4300 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42411
I20260924 03:02:50.090103 4428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42411 every 8 connection(s)
I20260924 03:02:50.090423 4300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:02:50.093884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 4300
I20260924 03:02:50.093959 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218948286999-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:02:50.094165 392 external_mini_cluster.cc:1428] Setting key 7c09d042f9839f20d041c53df4762bd1
Sep 24 03:02:50 dist-test-slave-wgsb krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:50.100463 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:50.091134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46281 (local address 127.0.98.62:40165)
0924 03:02:50.091180 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:50.091185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:50.091415 (+ 230us) server_negotiation.cc:407] Connection header received
0924 03:02:50.092435 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:50.092436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:50.092449 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:50.092514 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:50.093166 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:50.093667 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:50.095108 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:50.095413 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:50.095494 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:50.097716 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:50.097717 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:50.097717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:50.097724 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:50.099063 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:50.099494 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:50.099495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:50.099496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:50.099556 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:50.099825 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:50.099826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:50.099827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:50.099942 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:50.100023 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:50.100287 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 03:02:50.100371 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:02:50.100646 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:50.091117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40165 (local address 127.0.98.3:46281)
0924 03:02:50.091269 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:50.091312 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:50.092317 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:50.092583 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:50.092587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:50.092849 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:50.093064 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:50.093068 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:50.093770 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:50.093772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:50.094989 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:50.094992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:50.095507 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:50.095508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:50.095675 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:50.096297 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:50.096306 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:50.097610 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:50.099161 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:50.099168 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:50.099170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:50.099389 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:50.099641 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:50.099642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:50.099643 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:50.099754 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:50.100014 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:50.100020 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:50.100157 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 03:02:50.100267 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 03:02:50.100419 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260924 03:02:50.101065 4429 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40165
I20260924 03:02:50.101154 4429 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:50.101291 4429 heartbeater.cc:511] Master 127.0.98.62:40165 requested a full tablet report, sending...
I20260924 03:02:50.101646 3970 ts_manager.cc:194] Registered new tserver with Master: 25f2fc12a6d24997af8384c0c3c39413 (127.0.98.3:42411)
I20260924 03:02:50.101969 3970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46281
I20260924 03:02:50.106078 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 03:02:50 dist-test-slave-wgsb krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 03:02:50 dist-test-slave-wgsb krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 03:02:50.130527 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:50.122046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55168 (local address 127.0.98.1:46055)
0924 03:02:50.122250 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:50.122255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:50.122266 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:02:50.122293 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:50.122295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:50.122315 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:50.122388 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:50.123037 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:50.123864 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:50.124774 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:50.125099 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:50.125175 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:50.127513 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:50.127520 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:50.127521 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:50.127527 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:50.129302 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:50.129699 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:50.129702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:50.129702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:50.129768 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:50.129995 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:50.129995 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:50.129996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:50.130125 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:50.130254 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:50.130386 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 03:02:50.130449 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":91,"threads_started":1}
Sep 24 03:02:50 dist-test-slave-wgsb krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:50.138872 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:50.131961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35224 (local address 127.0.98.62:40165)
0924 03:02:50.132027 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:50.132030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:50.132073 (+ 43us) server_negotiation.cc:407] Connection header received
0924 03:02:50.132157 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:50.132159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:50.132170 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:50.132232 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:50.132766 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:50.133325 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:50.134095 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:50.134315 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:50.134408 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:50.136525 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:50.136527 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:50.136527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:50.136535 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:50.137692 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:50.138088 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:50.138089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:50.138089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:50.138167 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:50.138334 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:50.138334 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:50.138335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:50.138469 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:50.138544 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:50.138738 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 03:02:50.138800 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 03:02:50.139117 3970 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35224
I20260924 03:02:50.139720 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 4032
I20260924 03:02:50.144456 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 4166
I20260924 03:02:50.148335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 4300
I20260924 03:02:50.152325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 3938
2026-09-24T03:02:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1876 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:02:50 dist-test-slave-wgsb krb5kdc[4441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:02:50 dist-test-slave-wgsb krb5kdc[4441](info): set up 2 sockets
Sep 24 03:02:50 dist-test-slave-wgsb krb5kdc[4441](info): commencing operation
krb5kdc: starting...
W20260924 03:02:51.239830 392 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:02:51 dist-test-slave-wgsb krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:02:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:02:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:02:51.370000 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46255
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:51.450444 4454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:51.450654 4454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:51.450677 4454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:51.451794 4454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:02:51.451829 4454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:51.451843 4454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:51.451856 4454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:02:51.451869 4454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:02:51.453289 4454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:51.453474 4454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:51.453639 4454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:51.458436 4462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:51.458473 4459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:51.458575 4454 server_base.cc:1034] running on GCE node
W20260924 03:02:51.458477 4460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:51.458972 4454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:51.459354 4454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:51.460561 4454 hybrid_clock.cc:648] HybridClock initialized: now 1790218971460535 us; error 34 us; skew 500 ppm
Sep 24 03:02:51 dist-test-slave-wgsb krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:51.462970 4454 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:02:51.463382 4454 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:51.463454 4454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:51.463622 4454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:51.464650 4454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "1f158c85223e47ad9f79e84a799a3c17"
format_stamp: "Formatted at 2026-09-24 03:02:51 on dist-test-slave-wgsb"
server_key: "b649393b6a934c4bad00afa027bca7ce"
server_key_iv: "e694ca06e90d9d855181b9a26cee347b"
server_key_version: "encryptionkey@0"
I20260924 03:02:51.465019 4454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "1f158c85223e47ad9f79e84a799a3c17"
format_stamp: "Formatted at 2026-09-24 03:02:51 on dist-test-slave-wgsb"
server_key: "b649393b6a934c4bad00afa027bca7ce"
server_key_iv: "e694ca06e90d9d855181b9a26cee347b"
server_key_version: "encryptionkey@0"
I20260924 03:02:51.466331 4454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:02:51.467155 4468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:51.467334 4454 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:51.467432 4454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "1f158c85223e47ad9f79e84a799a3c17"
format_stamp: "Formatted at 2026-09-24 03:02:51 on dist-test-slave-wgsb"
server_key: "b649393b6a934c4bad00afa027bca7ce"
server_key_iv: "e694ca06e90d9d855181b9a26cee347b"
server_key_version: "encryptionkey@0"
I20260924 03:02:51.467511 4454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:51.477332 4454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:51.479316 4454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:51.479485 4454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:51.483676 4454 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46255
I20260924 03:02:51.483752 4520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46255 every 8 connection(s)
I20260924 03:02:51.484046 4454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:02:51.484653 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 4454
I20260924 03:02:51.484761 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:02:51.484969 392 external_mini_cluster.cc:1428] Setting key 9c63131140b96661872a858a0d968de4
I20260924 03:02:51.485054 4521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:51.488498 4521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f158c85223e47ad9f79e84a799a3c17: Bootstrap starting.
I20260924 03:02:51.489128 4521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f158c85223e47ad9f79e84a799a3c17: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:51.489454 4521 log.cc:824] T 00000000000000000000000000000000 P 1f158c85223e47ad9f79e84a799a3c17: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:51.490149 4521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f158c85223e47ad9f79e84a799a3c17: No bootstrap required, opened a new log
Sep 24 03:02:51 dist-test-slave-wgsb krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:51.492280 4521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f158c85223e47ad9f79e84a799a3c17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f158c85223e47ad9f79e84a799a3c17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46255 } }
I20260924 03:02:51.492461 4521 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f158c85223e47ad9f79e84a799a3c17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:51.492488 4521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f158c85223e47ad9f79e84a799a3c17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f158c85223e47ad9f79e84a799a3c17, State: Initialized, Role: FOLLOWER
I20260924 03:02:51.492632 4521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f158c85223e47ad9f79e84a799a3c17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f158c85223e47ad9f79e84a799a3c17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46255 } }
I20260924 03:02:51.492704 4521 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f158c85223e47ad9f79e84a799a3c17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:02:51.492727 4521 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f158c85223e47ad9f79e84a799a3c17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:02:51.492817 4521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f158c85223e47ad9f79e84a799a3c17 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:51.493505 4521 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f158c85223e47ad9f79e84a799a3c17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f158c85223e47ad9f79e84a799a3c17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46255 } }
I20260924 03:02:51.493626 4521 leader_election.cc:302] T 00000000000000000000000000000000 P 1f158c85223e47ad9f79e84a799a3c17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f158c85223e47ad9f79e84a799a3c17; no voters:
I20260924 03:02:51.493796 4521 leader_election.cc:288] T 00000000000000000000000000000000 P 1f158c85223e47ad9f79e84a799a3c17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:02:51.493848 4526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f158c85223e47ad9f79e84a799a3c17 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:51.494000 4526 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f158c85223e47ad9f79e84a799a3c17 [term 1 LEADER]: Becoming Leader. State: Replica: 1f158c85223e47ad9f79e84a799a3c17, State: Running, Role: LEADER
I20260924 03:02:51.494057 4521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f158c85223e47ad9f79e84a799a3c17 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:02:51.494174 4526 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f158c85223e47ad9f79e84a799a3c17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f158c85223e47ad9f79e84a799a3c17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46255 } }
I20260924 03:02:51.494565 4527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f158c85223e47ad9f79e84a799a3c17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f158c85223e47ad9f79e84a799a3c17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f158c85223e47ad9f79e84a799a3c17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46255 } } }
I20260924 03:02:51.494650 4527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f158c85223e47ad9f79e84a799a3c17 [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:51.494915 4528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f158c85223e47ad9f79e84a799a3c17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f158c85223e47ad9f79e84a799a3c17. Latest consensus state: current_term: 1 leader_uuid: "1f158c85223e47ad9f79e84a799a3c17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f158c85223e47ad9f79e84a799a3c17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46255 } } }
I20260924 03:02:51.495198 4528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f158c85223e47ad9f79e84a799a3c17 [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:51.495412 4533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:02:51.496047 4533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:02:51.496767 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:51.485523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34434 (local address 127.0.98.62:46255)
0924 03:02:51.485704 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:51.485713 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:51.485753 (+ 40us) server_negotiation.cc:407] Connection header received
0924 03:02:51.486331 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:51.486340 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:51.486401 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:51.486551 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:51.487135 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:51.488481 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:51.489293 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:51.489813 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:51.489934 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:51.492861 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:51.492865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:51.492866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:51.492896 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:51.494937 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:51.495390 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:51.495393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:51.495394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:51.495522 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:51.495790 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:51.495791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:51.495791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:51.496089 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:51.496270 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:51.496419 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 03:02:51.496555 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20260924 03:02:51.497625 4533 catalog_manager.cc:1347] Generated new cluster ID: e9d32fb75b924e4e9073fb9f2134fd09
I20260924 03:02:51.497694 4533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:02:51.515161 4533 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:02:51.515947 4533 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:02:51.521818 4533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f158c85223e47ad9f79e84a799a3c17: Generated new TSK 0
I20260924 03:02:51.522028 4533 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:02:51.552281 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46255
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:51.636032 4547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:51.636255 4547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:51.636281 4547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:51.637486 4547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:51.637533 4547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:51.637562 4547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:02:51.639060 4547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:51.639313 4547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:51.639585 4547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:51.644721 4553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:51.644724 4555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:51.644745 4552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:51.645069 4547 server_base.cc:1034] running on GCE node
I20260924 03:02:51.645378 4547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:51.645676 4547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:51.646829 4547 hybrid_clock.cc:648] HybridClock initialized: now 1790218971646817 us; error 27 us; skew 500 ppm
Sep 24 03:02:51 dist-test-slave-wgsb krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:51.649592 4547 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:02:51.649914 4547 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:51.649963 4547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:51.650103 4547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:51.651153 4547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-0/data/instance:
uuid: "b23d3a387c1041a0810b0d060028ae7f"
format_stamp: "Formatted at 2026-09-24 03:02:51 on dist-test-slave-wgsb"
server_key: "dd0d62be5b4712f286f4955be09c9a13"
server_key_iv: "fd4eb3180ef22a9c5cc770f5206bd76d"
server_key_version: "encryptionkey@0"
I20260924 03:02:51.651533 4547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b23d3a387c1041a0810b0d060028ae7f"
format_stamp: "Formatted at 2026-09-24 03:02:51 on dist-test-slave-wgsb"
server_key: "dd0d62be5b4712f286f4955be09c9a13"
server_key_iv: "fd4eb3180ef22a9c5cc770f5206bd76d"
server_key_version: "encryptionkey@0"
I20260924 03:02:51.652848 4547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:02:51.653596 4561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:51.653785 4547 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:02:51.653874 4547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "b23d3a387c1041a0810b0d060028ae7f"
format_stamp: "Formatted at 2026-09-24 03:02:51 on dist-test-slave-wgsb"
server_key: "dd0d62be5b4712f286f4955be09c9a13"
server_key_iv: "fd4eb3180ef22a9c5cc770f5206bd76d"
server_key_version: "encryptionkey@0"
I20260924 03:02:51.653925 4547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:51.671778 4547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:51.673645 4547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:51.673801 4547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:51.673985 4547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:51.674314 4547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:51.674350 4547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:51.674372 4547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:51.674420 4547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:51.680202 4547 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34959
I20260924 03:02:51.680287 4674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34959 every 8 connection(s)
I20260924 03:02:51.680555 4547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 03:02:51 dist-test-slave-wgsb krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:51.687335 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 4547
I20260924 03:02:51.687453 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:02:51.687688 392 external_mini_cluster.cc:1428] Setting key f7274894716d38d8acdebf71cab6b039
I20260924 03:02:51.690281 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:51.681283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54319 (local address 127.0.98.62:46255)
0924 03:02:51.681365 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:51.681367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:51.681545 (+ 178us) server_negotiation.cc:407] Connection header received
0924 03:02:51.682023 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:51.682025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:51.682035 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:51.682107 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:51.682970 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:51.683488 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:51.684548 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:51.684748 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:51.684805 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:51.686977 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:51.686979 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:51.686979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:51.686988 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:51.688845 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:51.689243 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:51.689244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:51.689245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:51.689309 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:51.689535 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:51.689536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:51.689536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:51.689690 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:51.689761 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:51.690130 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 03:02:51.690216 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 03:02:51.690532 4677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:51.681230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46255 (local address 127.0.98.1:54319)
0924 03:02:51.681421 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:51.681456 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:51.681917 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:51.682141 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:51.682144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:51.682399 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:51.682878 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:51.682889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:51.683571 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:51.683572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:51.684464 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:51.684466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:51.684824 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:51.684825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:51.684925 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:51.685449 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:51.685456 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:51.686870 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:51.688932 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:51.688938 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:51.688939 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:51.689166 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:51.689375 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:51.689375 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:51.689376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:51.689472 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:51.689761 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:51.689767 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:51.689947 (+ 180us) client_negotiation.cc:769] Sending connection context
0924 03:02:51.690073 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 03:02:51.690262 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":109,"threads_started":1}
I20260924 03:02:51.691068 4675 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46255
I20260924 03:02:51.691191 4675 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:51.691349 4675 heartbeater.cc:511] Master 127.0.98.62:46255 requested a full tablet report, sending...
I20260924 03:02:51.691959 4485 ts_manager.cc:194] Registered new tserver with Master: b23d3a387c1041a0810b0d060028ae7f (127.0.98.1:34959)
I20260924 03:02:51.692970 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54319
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:02:51.715969 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46255
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:51.796804 4680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:51.796998 4680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:51.797024 4680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:51.798175 4680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:51.798216 4680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:51.798243 4680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:02:51.799747 4680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:51.799983 4680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:51.800204 4680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:51.805066 4688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:51.805061 4685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:51.805061 4686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:51.805131 4680 server_base.cc:1034] running on GCE node
I20260924 03:02:51.805465 4680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:51.805661 4680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:51.806846 4680 hybrid_clock.cc:648] HybridClock initialized: now 1790218971806832 us; error 29 us; skew 500 ppm
Sep 24 03:02:51 dist-test-slave-wgsb krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:51.810046 4680 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:02:51.810444 4680 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:51.810516 4680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:51.810662 4680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:51.811853 4680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-1/data/instance:
uuid: "9d9d2835222f493188b912909d696037"
format_stamp: "Formatted at 2026-09-24 03:02:51 on dist-test-slave-wgsb"
server_key: "e709d629fef85723f3419110791307d7"
server_key_iv: "c8d989fa0090e2c2761f0113aa4a82f0"
server_key_version: "encryptionkey@0"
I20260924 03:02:51.812219 4680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9d9d2835222f493188b912909d696037"
format_stamp: "Formatted at 2026-09-24 03:02:51 on dist-test-slave-wgsb"
server_key: "e709d629fef85723f3419110791307d7"
server_key_iv: "c8d989fa0090e2c2761f0113aa4a82f0"
server_key_version: "encryptionkey@0"
I20260924 03:02:51.813570 4680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:02:51.814426 4694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:51.814692 4680 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:51.814800 4680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "9d9d2835222f493188b912909d696037"
format_stamp: "Formatted at 2026-09-24 03:02:51 on dist-test-slave-wgsb"
server_key: "e709d629fef85723f3419110791307d7"
server_key_iv: "c8d989fa0090e2c2761f0113aa4a82f0"
server_key_version: "encryptionkey@0"
I20260924 03:02:51.814896 4680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:51.833474 4680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:51.835716 4680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:51.835882 4680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:51.836124 4680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:51.836462 4680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:51.836519 4680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:51.836562 4680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:51.836588 4680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:51.842445 4680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37845
I20260924 03:02:51.842487 4807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37845 every 8 connection(s)
I20260924 03:02:51.842761 4680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:02:51.850850 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 4680
I20260924 03:02:51.850943 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:02:51.851181 392 external_mini_cluster.cc:1428] Setting key cd23fc03d4d27d09d96bbb3a53392dfd
Sep 24 03:02:51 dist-test-slave-wgsb krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:51.856199 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:51.843523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43989 (local address 127.0.98.62:46255)
0924 03:02:51.843748 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:51.843752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:51.844135 (+ 383us) server_negotiation.cc:407] Connection header received
0924 03:02:51.844842 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:51.844844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:51.844857 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:51.844924 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:51.846012 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:51.846598 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:51.848371 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:51.848608 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:51.848704 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:51.852653 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:51.852655 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:51.852656 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:51.852666 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:51.854496 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:51.855006 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:51.855008 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:51.855008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:51.855102 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:51.855360 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:51.855361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:51.855362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:51.855548 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:51.855640 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:51.855980 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 03:02:51.856086 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:02:51.856377 4810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:51.843474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46255 (local address 127.0.98.2:43989)
0924 03:02:51.843966 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:51.844016 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:51.844515 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:51.845023 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:51.845028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:51.845345 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:51.845851 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:51.845856 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:51.846786 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:51.846789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:51.847970 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:51.847976 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:51.848921 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:51.848922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:51.849086 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:51.850780 (+ 1694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:51.850790 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:51.852516 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:51.854612 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:51.854620 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:51.854622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:51.854862 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:51.855176 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:51.855177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:51.855178 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:51.855282 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:51.855685 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:51.855691 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:51.855826 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 03:02:51.855911 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 03:02:51.856050 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":371,"threads_started":1}
I20260924 03:02:51.856943 4808 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46255
I20260924 03:02:51.857064 4808 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:51.857241 4808 heartbeater.cc:511] Master 127.0.98.62:46255 requested a full tablet report, sending...
I20260924 03:02:51.857672 4485 ts_manager.cc:194] Registered new tserver with Master: 9d9d2835222f493188b912909d696037 (127.0.98.2:37845)
I20260924 03:02:51.858074 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43989
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:02:51.879853 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46255
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:51.961598 4813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:51.961843 4813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:51.961879 4813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:51.963057 4813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:51.963124 4813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:51.963169 4813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:02:51.964725 4813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:51.964986 4813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:51.965215 4813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:51.970579 4818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:51.970544 4821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:51.970629 4813 server_base.cc:1034] running on GCE node
W20260924 03:02:51.970638 4819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:51.971091 4813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:51.971340 4813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:51.972536 4813 hybrid_clock.cc:648] HybridClock initialized: now 1790218971972531 us; error 53 us; skew 500 ppm
Sep 24 03:02:51 dist-test-slave-wgsb krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:51.975245 4813 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:02:51.975703 4813 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:51.975782 4813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:51.975934 4813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:51.976992 4813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-2/data/instance:
uuid: "2c4fa5cf1ab74415bcbd1609f7bab635"
format_stamp: "Formatted at 2026-09-24 03:02:51 on dist-test-slave-wgsb"
server_key: "bf246e067a906eeb353a61c709d6fac5"
server_key_iv: "79d3b9a359b6f7f7b326a69e99e394f1"
server_key_version: "encryptionkey@0"
I20260924 03:02:51.977427 4813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2c4fa5cf1ab74415bcbd1609f7bab635"
format_stamp: "Formatted at 2026-09-24 03:02:51 on dist-test-slave-wgsb"
server_key: "bf246e067a906eeb353a61c709d6fac5"
server_key_iv: "79d3b9a359b6f7f7b326a69e99e394f1"
server_key_version: "encryptionkey@0"
I20260924 03:02:51.978767 4813 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:02:51.979683 4827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:51.979846 4813 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:51.979936 4813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "2c4fa5cf1ab74415bcbd1609f7bab635"
format_stamp: "Formatted at 2026-09-24 03:02:51 on dist-test-slave-wgsb"
server_key: "bf246e067a906eeb353a61c709d6fac5"
server_key_iv: "79d3b9a359b6f7f7b326a69e99e394f1"
server_key_version: "encryptionkey@0"
I20260924 03:02:51.980005 4813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:51.994315 4813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:51.996622 4813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:51.996757 4813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:51.997001 4813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:51.997318 4813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:51.997354 4813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:51.997376 4813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:51.997422 4813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:52.003031 4813 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46493
I20260924 03:02:52.003105 4940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46493 every 8 connection(s)
I20260924 03:02:52.003357 4813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:02:52.004758 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 4813
I20260924 03:02:52.004832 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:02:52.005015 392 external_mini_cluster.cc:1428] Setting key 950e442c50ba44c11f104bed23fcd0ef
Sep 24 03:02:52 dist-test-slave-wgsb krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:52.013017 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.004110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60131 (local address 127.0.98.62:46255)
0924 03:02:52.004153 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:52.004155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:52.004394 (+ 239us) server_negotiation.cc:407] Connection header received
0924 03:02:52.005008 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:52.005009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:52.005020 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:52.005077 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:52.006276 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.006799 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.007850 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.008050 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.008129 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:52.010346 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:52.010347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:52.010348 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:52.010354 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:52.011773 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:52.012156 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:52.012157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:52.012158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:52.012216 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:52.012429 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:52.012429 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:52.012430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:52.012533 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:52.012596 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:52.012885 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 03:02:52.012952 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:02:52.013330 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.004094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46255 (local address 127.0.98.3:60131)
0924 03:02:52.004267 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:52.004309 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:52.004894 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:52.005192 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:52.005196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:52.005563 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:52.006181 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:52.006186 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.006890 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.006891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:52.007768 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:52.007770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.008132 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.008133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:52.008265 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:52.008813 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:52.008821 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:52.010243 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:52.011861 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:52.011866 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:52.011868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:52.012072 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:52.012282 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:52.012282 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:52.012282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:52.012373 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:52.012654 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:52.012659 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:52.012770 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 03:02:52.012856 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 03:02:52.012969 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260924 03:02:52.013885 4941 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46255
I20260924 03:02:52.013986 4941 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:52.014148 4941 heartbeater.cc:511] Master 127.0.98.62:46255 requested a full tablet report, sending...
I20260924 03:02:52.014580 4485 ts_manager.cc:194] Registered new tserver with Master: 2c4fa5cf1ab74415bcbd1609f7bab635 (127.0.98.3:46493)
I20260924 03:02:52.014968 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60131
I20260924 03:02:52.016916 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:02:52.025121 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.018312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34442 (local address 127.0.98.62:46255)
0924 03:02:52.018387 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:52.018388 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:52.018519 (+ 131us) server_negotiation.cc:407] Connection header received
0924 03:02:52.018592 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:52.018593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:52.018606 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:52.018675 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:52.019342 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.019942 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.020688 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.020875 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.020946 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:52.022213 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:52.022215 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:52.022215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:52.022223 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:52.023875 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:52.024268 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:52.024269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:52.024270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:52.024330 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:52.024546 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:52.024547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:52.024547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:52.024707 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:52.024815 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:52.024936 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 03:02:52.025018 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 03:02:52.026826 4485 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:02:52.028581 4485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:02:52.036636 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.031103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37845 (local address 127.0.0.1:60976)
0924 03:02:52.031332 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:52.031412 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:52.031545 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:52.032751 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:52.032754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:52.032757 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:52.032833 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:52.032835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.034236 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.034237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:52.035630 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:52.035633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.036331 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.036332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:52.036411 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:52.036425 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 03:02:52.036494 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 03:02:52.036545 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":75,"threads_started":1}
I20260924 03:02:52.036638 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.031383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60976 (local address 127.0.98.2:37845)
0924 03:02:52.032568 (+ 1185us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:52.032572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:52.032582 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:02:52.032641 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:52.032643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:52.032659 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:52.032747 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:52.032924 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.034130 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.035728 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.036223 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.036328 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:52.036391 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:52.036505 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 03:02:52.036544 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1150,"thread_start_us":78,"threads_started":1}
I20260924 03:02:52.036903 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.031316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46493 (local address 127.0.0.1:41268)
0924 03:02:52.032849 (+ 1533us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:52.032860 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:52.032902 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:52.033174 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:52.033175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:52.033177 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:52.033219 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:52.033220 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.034510 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.034511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:52.035640 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:52.035642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.036651 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.036652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:52.036717 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:52.036721 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:02:52.036773 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 03:02:52.036804 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1513,"thread_start_us":58,"threads_started":1}
I20260924 03:02:52.036903 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.031110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34959 (local address 127.0.0.1:42124)
0924 03:02:52.031725 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:52.031736 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:52.031786 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:52.032962 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:52.032964 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:52.032966 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:52.033017 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:52.033019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.034506 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.034508 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:52.035501 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:52.035504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.036599 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.036600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:52.036671 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:52.036687 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 03:02:52.036724 (+ 37us) client_negotiation.cc:240] Negotiation successful
0924 03:02:52.036779 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":16,"thread_start_us":64,"threads_started":1}
I20260924 03:02:52.037163 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.032344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42124 (local address 127.0.98.1:34959)
0924 03:02:52.032766 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:52.032771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:52.032782 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:02:52.032824 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:52.032826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:52.032843 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:52.033101 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:52.033185 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.034369 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.035681 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.036477 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.036933 (+ 456us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:52.037003 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:52.037031 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 03:02:52.037077 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":103,"threads_started":1}
I20260924 03:02:52.037170 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.031503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41268 (local address 127.0.98.3:46493)
0924 03:02:52.033020 (+ 1517us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:52.033024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:52.033032 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:02:52.033066 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:52.033068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:52.033082 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:52.033294 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:52.033367 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.034404 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.035709 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.036485 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.036560 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:52.036622 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:52.037043 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 03:02:52.037089 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1480,"thread_start_us":70,"threads_started":1}
I20260924 03:02:52.038025 4875 tablet_service.cc:1467] Processing CreateTablet for tablet c6cf97fbf0e34c2e86eed2c51a567ea8 (DEFAULT_TABLE table=test-table [id=ba35aed17281427aa75867f52cb9b9bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:52.038019 4609 tablet_service.cc:1467] Processing CreateTablet for tablet c6cf97fbf0e34c2e86eed2c51a567ea8 (DEFAULT_TABLE table=test-table [id=ba35aed17281427aa75867f52cb9b9bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:52.038275 4742 tablet_service.cc:1467] Processing CreateTablet for tablet c6cf97fbf0e34c2e86eed2c51a567ea8 (DEFAULT_TABLE table=test-table [id=ba35aed17281427aa75867f52cb9b9bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:52.038414 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6cf97fbf0e34c2e86eed2c51a567ea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:52.038414 4875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6cf97fbf0e34c2e86eed2c51a567ea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:52.038620 4742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6cf97fbf0e34c2e86eed2c51a567ea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:52.042163 4958 tablet_bootstrap.cc:492] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 9d9d2835222f493188b912909d696037: Bootstrap starting.
I20260924 03:02:52.042766 4958 tablet_bootstrap.cc:654] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 9d9d2835222f493188b912909d696037: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:52.043286 4958 log.cc:824] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 9d9d2835222f493188b912909d696037: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:52.043692 4960 tablet_bootstrap.cc:492] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 2c4fa5cf1ab74415bcbd1609f7bab635: Bootstrap starting.
I20260924 03:02:52.044292 4958 tablet_bootstrap.cc:492] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 9d9d2835222f493188b912909d696037: No bootstrap required, opened a new log
I20260924 03:02:52.044304 4960 tablet_bootstrap.cc:654] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 2c4fa5cf1ab74415bcbd1609f7bab635: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:52.044368 4958 ts_tablet_manager.cc:1397] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 9d9d2835222f493188b912909d696037: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:02:52.044909 4960 log.cc:824] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 2c4fa5cf1ab74415bcbd1609f7bab635: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:52.045260 4959 tablet_bootstrap.cc:492] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f: Bootstrap starting.
I20260924 03:02:52.045584 4960 tablet_bootstrap.cc:492] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 2c4fa5cf1ab74415bcbd1609f7bab635: No bootstrap required, opened a new log
I20260924 03:02:52.045665 4960 ts_tablet_manager.cc:1397] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 2c4fa5cf1ab74415bcbd1609f7bab635: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 03:02:52.045862 4959 tablet_bootstrap.cc:654] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:52.046425 4959 log.cc:824] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:52.046648 4958 raft_consensus.cc:348] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 9d9d2835222f493188b912909d696037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23d3a387c1041a0810b0d060028ae7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34959 } } peers { permanent_uuid: "9d9d2835222f493188b912909d696037" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37845 } } peers { permanent_uuid: "2c4fa5cf1ab74415bcbd1609f7bab635" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46493 } }
I20260924 03:02:52.046792 4958 raft_consensus.cc:374] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 9d9d2835222f493188b912909d696037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:52.046836 4958 raft_consensus.cc:729] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 9d9d2835222f493188b912909d696037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d9d2835222f493188b912909d696037, State: Initialized, Role: FOLLOWER
I20260924 03:02:52.046979 4958 consensus_queue.cc:260] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 9d9d2835222f493188b912909d696037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23d3a387c1041a0810b0d060028ae7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34959 } } peers { permanent_uuid: "9d9d2835222f493188b912909d696037" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37845 } } peers { permanent_uuid: "2c4fa5cf1ab74415bcbd1609f7bab635" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46493 } }
I20260924 03:02:52.047255 4959 tablet_bootstrap.cc:492] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f: No bootstrap required, opened a new log
I20260924 03:02:52.047276 4958 ts_tablet_manager.cc:1428] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 9d9d2835222f493188b912909d696037: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:02:52.047323 4959 ts_tablet_manager.cc:1397] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:02:52.047389 4808 heartbeater.cc:503] Master 127.0.98.62:46255 was elected leader, sending a full tablet report...
I20260924 03:02:52.047456 4960 raft_consensus.cc:348] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 2c4fa5cf1ab74415bcbd1609f7bab635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23d3a387c1041a0810b0d060028ae7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34959 } } peers { permanent_uuid: "9d9d2835222f493188b912909d696037" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37845 } } peers { permanent_uuid: "2c4fa5cf1ab74415bcbd1609f7bab635" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46493 } }
I20260924 03:02:52.047632 4960 raft_consensus.cc:374] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 2c4fa5cf1ab74415bcbd1609f7bab635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:52.047689 4960 raft_consensus.cc:729] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 2c4fa5cf1ab74415bcbd1609f7bab635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4fa5cf1ab74415bcbd1609f7bab635, State: Initialized, Role: FOLLOWER
I20260924 03:02:52.047803 4960 consensus_queue.cc:260] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 2c4fa5cf1ab74415bcbd1609f7bab635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23d3a387c1041a0810b0d060028ae7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34959 } } peers { permanent_uuid: "9d9d2835222f493188b912909d696037" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37845 } } peers { permanent_uuid: "2c4fa5cf1ab74415bcbd1609f7bab635" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46493 } }
I20260924 03:02:52.048046 4960 ts_tablet_manager.cc:1428] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 2c4fa5cf1ab74415bcbd1609f7bab635: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 03:02:52.048261 4941 heartbeater.cc:503] Master 127.0.98.62:46255 was elected leader, sending a full tablet report...
I20260924 03:02:52.049376 4959 raft_consensus.cc:348] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23d3a387c1041a0810b0d060028ae7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34959 } } peers { permanent_uuid: "9d9d2835222f493188b912909d696037" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37845 } } peers { permanent_uuid: "2c4fa5cf1ab74415bcbd1609f7bab635" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46493 } }
I20260924 03:02:52.049490 4959 raft_consensus.cc:374] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:52.049510 4959 raft_consensus.cc:729] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b23d3a387c1041a0810b0d060028ae7f, State: Initialized, Role: FOLLOWER
I20260924 03:02:52.049621 4959 consensus_queue.cc:260] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23d3a387c1041a0810b0d060028ae7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34959 } } peers { permanent_uuid: "9d9d2835222f493188b912909d696037" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37845 } } peers { permanent_uuid: "2c4fa5cf1ab74415bcbd1609f7bab635" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46493 } }
I20260924 03:02:52.049839 4959 ts_tablet_manager.cc:1428] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 03:02:52.049902 4675 heartbeater.cc:503] Master 127.0.98.62:46255 was elected leader, sending a full tablet report...
W20260924 03:02:52.093657 4809 tablet.cc:2369] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 9d9d2835222f493188b912909d696037: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:02:52.181295 4676 tablet.cc:2369] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:02:52.254035 4942 tablet.cc:2369] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 2c4fa5cf1ab74415bcbd1609f7bab635: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:02:52.341813 4966 raft_consensus.cc:482] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:02:52.341982 4966 raft_consensus.cc:504] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23d3a387c1041a0810b0d060028ae7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34959 } } peers { permanent_uuid: "9d9d2835222f493188b912909d696037" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37845 } } peers { permanent_uuid: "2c4fa5cf1ab74415bcbd1609f7bab635" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46493 } }
I20260924 03:02:52.342368 4966 leader_election.cc:288] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d9d2835222f493188b912909d696037 (127.0.98.2:37845), 2c4fa5cf1ab74415bcbd1609f7bab635 (127.0.98.3:46493)
I20260924 03:02:52.345574 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.342920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56503 (local address 127.0.98.3:46493)
0924 03:02:52.342965 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:52.342967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:52.342997 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:02:52.343096 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:52.343096 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:52.343105 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:52.343182 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:52.343352 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.343884 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.344829 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.345197 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.345304 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:52.345314 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:52.345464 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 03:02:52.345500 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 03:02:52.345578 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.342725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37643 (local address 127.0.98.2:37845)
0924 03:02:52.342819 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:52.342821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:52.342904 (+ 83us) server_negotiation.cc:407] Connection header received
0924 03:02:52.342968 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:52.342969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:52.342980 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:52.343054 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:52.343194 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.343754 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.344823 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.345204 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.345315 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:52.345324 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:52.345471 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 03:02:52.345507 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 03:02:52.345610 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.342607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37845 (local address 127.0.98.1:37643)
0924 03:02:52.342799 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:52.342812 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:52.342895 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:52.343059 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:52.343060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:52.343062 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:52.343120 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:52.343122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.343833 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.343834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:52.344727 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:52.344730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.345323 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.345324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:52.345387 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:52.345390 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:02:52.345451 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 03:02:52.345507 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":5,"thread_start_us":90,"threads_started":1}
I20260924 03:02:52.345625 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.342804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46493 (local address 127.0.98.1:56503)
0924 03:02:52.342918 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:52.342930 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:52.343009 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:52.343189 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:52.343189 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:52.343191 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:52.343265 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:52.343267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.343964 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.343965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:52.344728 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:52.344730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.345308 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.345308 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:52.345387 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:52.345390 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:02:52.345459 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 03:02:52.345499 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260924 03:02:52.345989 4762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6cf97fbf0e34c2e86eed2c51a567ea8" candidate_uuid: "b23d3a387c1041a0810b0d060028ae7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d9d2835222f493188b912909d696037" is_pre_election: true
I20260924 03:02:52.346081 4895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6cf97fbf0e34c2e86eed2c51a567ea8" candidate_uuid: "b23d3a387c1041a0810b0d060028ae7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4fa5cf1ab74415bcbd1609f7bab635" is_pre_election: true
I20260924 03:02:52.346170 4762 raft_consensus.cc:2454] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 9d9d2835222f493188b912909d696037 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b23d3a387c1041a0810b0d060028ae7f in term 0.
I20260924 03:02:52.346202 4895 raft_consensus.cc:2454] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 2c4fa5cf1ab74415bcbd1609f7bab635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b23d3a387c1041a0810b0d060028ae7f in term 0.
I20260924 03:02:52.346412 4565 leader_election.cc:302] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4fa5cf1ab74415bcbd1609f7bab635, b23d3a387c1041a0810b0d060028ae7f; no voters:
I20260924 03:02:52.346532 4966 raft_consensus.cc:2781] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:02:52.346629 4966 raft_consensus.cc:482] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:02:52.346666 4966 raft_consensus.cc:3037] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:52.347303 4966 raft_consensus.cc:504] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23d3a387c1041a0810b0d060028ae7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34959 } } peers { permanent_uuid: "9d9d2835222f493188b912909d696037" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37845 } } peers { permanent_uuid: "2c4fa5cf1ab74415bcbd1609f7bab635" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46493 } }
I20260924 03:02:52.347440 4966 leader_election.cc:288] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f [CANDIDATE]: Term 1 election: Requested vote from peers 9d9d2835222f493188b912909d696037 (127.0.98.2:37845), 2c4fa5cf1ab74415bcbd1609f7bab635 (127.0.98.3:46493)
I20260924 03:02:52.347683 4895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6cf97fbf0e34c2e86eed2c51a567ea8" candidate_uuid: "b23d3a387c1041a0810b0d060028ae7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4fa5cf1ab74415bcbd1609f7bab635"
I20260924 03:02:52.347666 4762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6cf97fbf0e34c2e86eed2c51a567ea8" candidate_uuid: "b23d3a387c1041a0810b0d060028ae7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d9d2835222f493188b912909d696037"
I20260924 03:02:52.347752 4762 raft_consensus.cc:3037] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 9d9d2835222f493188b912909d696037 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:52.347754 4895 raft_consensus.cc:3037] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 2c4fa5cf1ab74415bcbd1609f7bab635 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:52.348574 4762 raft_consensus.cc:2454] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 9d9d2835222f493188b912909d696037 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b23d3a387c1041a0810b0d060028ae7f in term 1.
I20260924 03:02:52.348574 4895 raft_consensus.cc:2454] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 2c4fa5cf1ab74415bcbd1609f7bab635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b23d3a387c1041a0810b0d060028ae7f in term 1.
I20260924 03:02:52.348790 4565 leader_election.cc:302] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d9d2835222f493188b912909d696037, b23d3a387c1041a0810b0d060028ae7f; no voters:
I20260924 03:02:52.348878 4966 raft_consensus.cc:2781] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:52.349014 4966 raft_consensus.cc:686] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f [term 1 LEADER]: Becoming Leader. State: Replica: b23d3a387c1041a0810b0d060028ae7f, State: Running, Role: LEADER
I20260924 03:02:52.349112 4966 consensus_queue.cc:237] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23d3a387c1041a0810b0d060028ae7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34959 } } peers { permanent_uuid: "9d9d2835222f493188b912909d696037" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37845 } } peers { permanent_uuid: "2c4fa5cf1ab74415bcbd1609f7bab635" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46493 } }
I20260924 03:02:52.349917 4483 catalog_manager.cc:5495] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f reported cstate change: term changed from 0 to 1, leader changed from <none> to b23d3a387c1041a0810b0d060028ae7f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b23d3a387c1041a0810b0d060028ae7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b23d3a387c1041a0810b0d060028ae7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d9d2835222f493188b912909d696037" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4fa5cf1ab74415bcbd1609f7bab635" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46493 } health_report { overall_health: UNKNOWN } } }
I20260924 03:02:52.389910 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.387507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42138 (local address 127.0.98.1:34959)
0924 03:02:52.387578 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:52.387580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:52.387608 (+ 28us) server_negotiation.cc:407] Connection header received
0924 03:02:52.387648 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:52.387649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:52.387672 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:52.387750 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:02:52.387875 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.388405 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.389188 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:52.389396 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:52.389465 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:52.389621 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:02:52.389730 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:02:52.389805 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:52.389824 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 03:02:52.389844 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 03:02:52.391849 4895 raft_consensus.cc:1261] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 2c4fa5cf1ab74415bcbd1609f7bab635 [term 1 FOLLOWER]: Refusing update from remote peer b23d3a387c1041a0810b0d060028ae7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:02:52.391894 4762 raft_consensus.cc:1261] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 9d9d2835222f493188b912909d696037 [term 1 FOLLOWER]: Refusing update from remote peer b23d3a387c1041a0810b0d060028ae7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:02:52.392386 4966 consensus_queue.cc:1035] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c4fa5cf1ab74415bcbd1609f7bab635" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:02:52.392570 4969 consensus_queue.cc:1035] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d9d2835222f493188b912909d696037" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:02:52.394827 4972 mvcc.cc:205] Tried to move back new op lower bound from 7332736910913986560 to 7332736910742421504. Current Snapshot: MvccSnapshot[applied={T|T < 7332736910913986560}]
I20260924 03:02:52.395996 4974 mvcc.cc:205] Tried to move back new op lower bound from 7332736910913986560 to 7332736910742421504. Current Snapshot: MvccSnapshot[applied={T|T < 7332736910913986560 or (T in {7332736910913986560})}]
I20260924 03:02:52.396559 4973 mvcc.cc:205] Tried to move back new op lower bound from 7332736910913986560 to 7332736910742421504. Current Snapshot: MvccSnapshot[applied={T|T < 7332736910913986560}]
I20260924 03:02:52.397629 4484 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:02:52.397742 4484 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:02:52.398576 4484 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f158c85223e47ad9f79e84a799a3c17: Sending DeleteTablet for 3 replicas of tablet c6cf97fbf0e34c2e86eed2c51a567ea8
I20260924 03:02:52.398897 4609 tablet_service.cc:1514] Processing DeleteTablet for tablet c6cf97fbf0e34c2e86eed2c51a567ea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:02:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42124
I20260924 03:02:52.398914 4742 tablet_service.cc:1514] Processing DeleteTablet for tablet c6cf97fbf0e34c2e86eed2c51a567ea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:02:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60976
I20260924 03:02:52.399183 4983 tablet_replica.cc:330] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 9d9d2835222f493188b912909d696037: stopping tablet replica
I20260924 03:02:52.399170 4875 tablet_service.cc:1514] Processing DeleteTablet for tablet c6cf97fbf0e34c2e86eed2c51a567ea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:02:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41268
I20260924 03:02:52.399284 4983 raft_consensus.cc:2229] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 9d9d2835222f493188b912909d696037 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:02:52.399375 4983 raft_consensus.cc:2258] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 9d9d2835222f493188b912909d696037 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:02:52.399369 4982 tablet_replica.cc:330] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f: stopping tablet replica
I20260924 03:02:52.399434 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 4547
I20260924 03:02:52.399458 4982 raft_consensus.cc:2229] T c6cf97fbf0e34c2e86eed2c51a567ea8 P b23d3a387c1041a0810b0d060028ae7f [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:02:52.400449 4983 ts_tablet_manager.cc:1905] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 9d9d2835222f493188b912909d696037: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:02:52.402082 4983 ts_tablet_manager.cc:1918] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 9d9d2835222f493188b912909d696037: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:02:52.402132 4983 log.cc:1196] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 9d9d2835222f493188b912909d696037: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-1/wal/wals/c6cf97fbf0e34c2e86eed2c51a567ea8
I20260924 03:02:52.402387 4983 ts_tablet_manager.cc:1939] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 9d9d2835222f493188b912909d696037: Deleting consensus metadata
I20260924 03:02:52.402767 4472 catalog_manager.cc:4841] TS 9d9d2835222f493188b912909d696037 (127.0.98.2:37845): tablet c6cf97fbf0e34c2e86eed2c51a567ea8 (table test-table [id=ba35aed17281427aa75867f52cb9b9bf]) successfully deleted
I20260924 03:02:52.403849 4984 tablet_replica.cc:330] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 2c4fa5cf1ab74415bcbd1609f7bab635: stopping tablet replica
I20260924 03:02:52.403967 4984 raft_consensus.cc:2229] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 2c4fa5cf1ab74415bcbd1609f7bab635 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:02:52.404098 4984 raft_consensus.cc:2258] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 2c4fa5cf1ab74415bcbd1609f7bab635 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:02:52.404500 4984 ts_tablet_manager.cc:1905] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 2c4fa5cf1ab74415bcbd1609f7bab635: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 03:02:52.405552 4471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34959 (error 108)
I20260924 03:02:52.405932 4984 ts_tablet_manager.cc:1918] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 2c4fa5cf1ab74415bcbd1609f7bab635: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:02:52.406015 4984 log.cc:1196] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 2c4fa5cf1ab74415bcbd1609f7bab635: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218948286999-392-0/minicluster-data/ts-2/wal/wals/c6cf97fbf0e34c2e86eed2c51a567ea8
I20260924 03:02:52.406168 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.405979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34959 (local address 127.0.0.1:42140)
0924 03:02:52.406090 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:52.406128 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 03:02:52.406153 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 4680
W20260924 03:02:52.406286 4471 catalog_manager.cc:4584] TS b23d3a387c1041a0810b0d060028ae7f (127.0.98.1:34959): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c6cf97fbf0e34c2e86eed2c51a567ea8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34959: connect: Connection refused (error 111)
I20260924 03:02:52.406327 4984 ts_tablet_manager.cc:1939] T c6cf97fbf0e34c2e86eed2c51a567ea8 P 2c4fa5cf1ab74415bcbd1609f7bab635: Deleting consensus metadata
I20260924 03:02:52.407863 4472 catalog_manager.cc:4841] TS 2c4fa5cf1ab74415bcbd1609f7bab635 (127.0.98.3:46493): tablet c6cf97fbf0e34c2e86eed2c51a567ea8 (table test-table [id=ba35aed17281427aa75867f52cb9b9bf]) successfully deleted
I20260924 03:02:52.410844 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 4813
I20260924 03:02:52.415364 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 4454
2026-09-24T03:02:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2263 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T03:02:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:02:52Z Disabled control of system clock
I20260924 03:02:52.442414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37809
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 03:02:52.528715 4989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:52.529002 4989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:52.529039 4989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:52.530243 4989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:02:52.530311 4989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:52.530334 4989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:52.530365 4989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:02:52.530391 4989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:02:52.531847 4989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37809
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:52.532104 4989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:52.532393 4989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:52.537654 4994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:52.537493 4995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:52.537655 4989 server_base.cc:1034] running on GCE node
W20260924 03:02:52.537488 4997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:52.538156 4989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:52.538451 4989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:52.539664 4989 hybrid_clock.cc:648] HybridClock initialized: now 1790218972539677 us; error 75 us; skew 500 ppm
I20260924 03:02:52.541004 4989 webserver.cc:460] Webserver started at http://127.0.98.62:40245/ using document root <none> and password file <none>
I20260924 03:02:52.541265 4989 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:52.541332 4989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:52.541473 4989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:52.542523 4989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "8f19bb3814ea40c58a9daffbb2719881"
format_stamp: "Formatted at 2026-09-24 03:02:52 on dist-test-slave-wgsb"
server_key: "f6713db70693a0774b01f7d146f61caf"
server_key_iv: "627fcf10ac31aa828ec3856628fd249a"
server_key_version: "encryptionkey@0"
I20260924 03:02:52.542872 4989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "8f19bb3814ea40c58a9daffbb2719881"
format_stamp: "Formatted at 2026-09-24 03:02:52 on dist-test-slave-wgsb"
server_key: "f6713db70693a0774b01f7d146f61caf"
server_key_iv: "627fcf10ac31aa828ec3856628fd249a"
server_key_version: "encryptionkey@0"
I20260924 03:02:52.544252 4989 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:02:52.545009 5003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:52.545378 4989 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:52.545509 4989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "8f19bb3814ea40c58a9daffbb2719881"
format_stamp: "Formatted at 2026-09-24 03:02:52 on dist-test-slave-wgsb"
server_key: "f6713db70693a0774b01f7d146f61caf"
server_key_iv: "627fcf10ac31aa828ec3856628fd249a"
server_key_version: "encryptionkey@0"
I20260924 03:02:52.545595 4989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:52.549108 4989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:52.549361 4989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:52.549494 4989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:52.553200 4989 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37809
I20260924 03:02:52.553231 5055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37809 every 8 connection(s)
I20260924 03:02:52.553545 4989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:02:52.554138 5056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:52.556777 5056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f19bb3814ea40c58a9daffbb2719881: Bootstrap starting.
I20260924 03:02:52.557281 5056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f19bb3814ea40c58a9daffbb2719881: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:52.557511 5056 log.cc:824] T 00000000000000000000000000000000 P 8f19bb3814ea40c58a9daffbb2719881: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:52.558142 5056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f19bb3814ea40c58a9daffbb2719881: No bootstrap required, opened a new log
I20260924 03:02:52.558419 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 4989
I20260924 03:02:52.558509 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:02:52.558688 392 external_mini_cluster.cc:1428] Setting key dc5b179d2cb98a5d612bddfb6cdc3685
I20260924 03:02:52.560514 5056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f19bb3814ea40c58a9daffbb2719881 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f19bb3814ea40c58a9daffbb2719881" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37809 } }
I20260924 03:02:52.560694 5056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f19bb3814ea40c58a9daffbb2719881 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:52.560730 5056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f19bb3814ea40c58a9daffbb2719881 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f19bb3814ea40c58a9daffbb2719881, State: Initialized, Role: FOLLOWER
I20260924 03:02:52.560878 5056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f19bb3814ea40c58a9daffbb2719881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f19bb3814ea40c58a9daffbb2719881" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37809 } }
I20260924 03:02:52.560966 5056 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f19bb3814ea40c58a9daffbb2719881 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:02:52.561008 5056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f19bb3814ea40c58a9daffbb2719881 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:02:52.561038 5056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f19bb3814ea40c58a9daffbb2719881 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:52.561831 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.559201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37278 (local address 127.0.98.62:37809)
0924 03:02:52.559410 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:52.559420 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:52.559453 (+ 33us) server_negotiation.cc:407] Connection header received
0924 03:02:52.560466 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:52.560476 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:52.560529 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:52.560687 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:52.561402 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:52.561406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:52.561406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:02:52.561411 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:52.561458 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:02:52.561468 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:52.561545 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:52.561612 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 03:02:52.561677 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260924 03:02:52.561977 5056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f19bb3814ea40c58a9daffbb2719881 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f19bb3814ea40c58a9daffbb2719881" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37809 } }
I20260924 03:02:52.562086 5056 leader_election.cc:302] T 00000000000000000000000000000000 P 8f19bb3814ea40c58a9daffbb2719881 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f19bb3814ea40c58a9daffbb2719881; no voters:
I20260924 03:02:52.562258 5056 leader_election.cc:288] T 00000000000000000000000000000000 P 8f19bb3814ea40c58a9daffbb2719881 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:02:52.562289 5061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f19bb3814ea40c58a9daffbb2719881 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:52.562435 5061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f19bb3814ea40c58a9daffbb2719881 [term 1 LEADER]: Becoming Leader. State: Replica: 8f19bb3814ea40c58a9daffbb2719881, State: Running, Role: LEADER
I20260924 03:02:52.562529 5061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f19bb3814ea40c58a9daffbb2719881 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f19bb3814ea40c58a9daffbb2719881" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37809 } }
I20260924 03:02:52.562536 5056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f19bb3814ea40c58a9daffbb2719881 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:02:52.562899 5062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f19bb3814ea40c58a9daffbb2719881 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f19bb3814ea40c58a9daffbb2719881" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f19bb3814ea40c58a9daffbb2719881" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37809 } } }
I20260924 03:02:52.562983 5062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f19bb3814ea40c58a9daffbb2719881 [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:52.563671 5063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f19bb3814ea40c58a9daffbb2719881 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f19bb3814ea40c58a9daffbb2719881. Latest consensus state: current_term: 1 leader_uuid: "8f19bb3814ea40c58a9daffbb2719881" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f19bb3814ea40c58a9daffbb2719881" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37809 } } }
I20260924 03:02:52.563771 5063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f19bb3814ea40c58a9daffbb2719881 [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:52.563930 5067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:02:52.564430 5067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:02:52.565735 5067 catalog_manager.cc:1347] Generated new cluster ID: 0c78bcbf6f354529a5f0f9da99bd9147
I20260924 03:02:52.565785 5067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:02:52.584950 5067 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:02:52.585158 5067 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:02:52.587559 5067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f19bb3814ea40c58a9daffbb2719881: Generated new TSK 0
I20260924 03:02:52.587850 5067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:02:52.592742 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37809
--builtin_ntp_servers=127.0.98.20:32875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 03:02:52.676367 5080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:52.676620 5080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:52.676671 5080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:52.677848 5080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:52.677913 5080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:52.677956 5080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:02:52.679438 5080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:52.679733 5080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:52.680056 5080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:52.686018 5088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:52.686028 5086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:52.686136 5080 server_base.cc:1034] running on GCE node
W20260924 03:02:52.686177 5085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:52.686510 5080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:52.686784 5080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:52.687942 5080 hybrid_clock.cc:648] HybridClock initialized: now 1790218972687918 us; error 37 us; skew 500 ppm
I20260924 03:02:52.689316 5080 webserver.cc:460] Webserver started at http://127.0.98.1:42371/ using document root <none> and password file <none>
I20260924 03:02:52.689570 5080 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:52.689621 5080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:52.689752 5080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:52.690860 5080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-0/data/instance:
uuid: "dca1ff11088a46fe8e43b6b701377131"
format_stamp: "Formatted at 2026-09-24 03:02:52 on dist-test-slave-wgsb"
server_key: "717f050670adff8bb1cd40430647e319"
server_key_iv: "fe0e60185a09ee20d6d573d13f61e9eb"
server_key_version: "encryptionkey@0"
I20260924 03:02:52.691248 5080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dca1ff11088a46fe8e43b6b701377131"
format_stamp: "Formatted at 2026-09-24 03:02:52 on dist-test-slave-wgsb"
server_key: "717f050670adff8bb1cd40430647e319"
server_key_iv: "fe0e60185a09ee20d6d573d13f61e9eb"
server_key_version: "encryptionkey@0"
I20260924 03:02:52.692749 5080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:02:52.693606 5094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:52.693822 5080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:02:52.693935 5080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "dca1ff11088a46fe8e43b6b701377131"
format_stamp: "Formatted at 2026-09-24 03:02:52 on dist-test-slave-wgsb"
server_key: "717f050670adff8bb1cd40430647e319"
server_key_iv: "fe0e60185a09ee20d6d573d13f61e9eb"
server_key_version: "encryptionkey@0"
I20260924 03:02:52.694010 5080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:52.697305 5080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:52.697517 5080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:52.697616 5080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:52.697767 5080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:52.698001 5080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:52.698030 5080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:52.698050 5080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:52.698063 5080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:52.703871 5080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37393
I20260924 03:02:52.703974 5207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37393 every 8 connection(s)
I20260924 03:02:52.704276 5080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:02:52.708601 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 5080
I20260924 03:02:52.708696 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:02:52.708878 392 external_mini_cluster.cc:1428] Setting key 5b552f2c5a87d5a19be76a692c6dc933
I20260924 03:02:52.709911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37809
--builtin_ntp_servers=127.0.98.20:32875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 03:02:52.711272 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.708347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39741 (local address 127.0.98.62:37809)
0924 03:02:52.708487 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:52.708489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:52.708740 (+ 251us) server_negotiation.cc:407] Connection header received
0924 03:02:52.709458 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:52.709459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:52.709463 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:52.709552 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:52.710953 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:52.710955 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:52.710955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:02:52.710960 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:52.710976 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:02:52.710981 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:52.711056 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:52.711178 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 03:02:52.711195 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 03:02:52.711486 5210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.708205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37809 (local address 127.0.98.1:39741)
0924 03:02:52.708593 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:52.708630 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:52.709357 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:52.709656 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:52.709662 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:52.709673 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:52.710777 (+ 1104us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:02:52.710778 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:52.710808 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:02:52.710839 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:52.711068 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:52.711074 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:52.711077 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:02:52.711188 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 03:02:52.711253 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":135,"threads_started":1}
I20260924 03:02:52.712033 5208 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37809
I20260924 03:02:52.712136 5208 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:52.712319 5208 heartbeater.cc:511] Master 127.0.98.62:37809 requested a full tablet report, sending...
I20260924 03:02:52.712914 5020 ts_manager.cc:194] Registered new tserver with Master: dca1ff11088a46fe8e43b6b701377131 (127.0.98.1:37393)
W20260924 03:02:52.794694 5211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:52.794957 5211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:52.794982 5211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:52.796186 5211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:52.796228 5211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:52.796254 5211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:02:52.797719 5211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:52.797906 5211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:52.798147 5211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:52.803932 5216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:52.803936 5217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:52.803936 5219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:52.804205 5211 server_base.cc:1034] running on GCE node
I20260924 03:02:52.804478 5211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:52.804776 5211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:52.806013 5211 hybrid_clock.cc:648] HybridClock initialized: now 1790218972805993 us; error 36 us; skew 500 ppm
I20260924 03:02:52.807485 5211 webserver.cc:460] Webserver started at http://127.0.98.2:40087/ using document root <none> and password file <none>
I20260924 03:02:52.807767 5211 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:52.807834 5211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:52.807971 5211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:52.809103 5211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-1/data/instance:
uuid: "122d701c49594c23aeadc91b695fa723"
format_stamp: "Formatted at 2026-09-24 03:02:52 on dist-test-slave-wgsb"
server_key: "406ec64b614e06fa3125985a7883969f"
server_key_iv: "e77cd28ea4484be9b251870c69b21599"
server_key_version: "encryptionkey@0"
I20260924 03:02:52.809484 5211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-1/wal/instance:
uuid: "122d701c49594c23aeadc91b695fa723"
format_stamp: "Formatted at 2026-09-24 03:02:52 on dist-test-slave-wgsb"
server_key: "406ec64b614e06fa3125985a7883969f"
server_key_iv: "e77cd28ea4484be9b251870c69b21599"
server_key_version: "encryptionkey@0"
I20260924 03:02:52.810842 5211 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:02:52.811743 5225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:52.811924 5211 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:52.812031 5211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "122d701c49594c23aeadc91b695fa723"
format_stamp: "Formatted at 2026-09-24 03:02:52 on dist-test-slave-wgsb"
server_key: "406ec64b614e06fa3125985a7883969f"
server_key_iv: "e77cd28ea4484be9b251870c69b21599"
server_key_version: "encryptionkey@0"
I20260924 03:02:52.812121 5211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:52.815681 5211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:52.815955 5211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:52.816103 5211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:52.816339 5211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:52.816754 5211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:52.816802 5211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:52.816840 5211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:52.816864 5211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:52.822745 5211 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41989
I20260924 03:02:52.822824 5338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41989 every 8 connection(s)
I20260924 03:02:52.823146 5211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:02:52.825413 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 5211
I20260924 03:02:52.825639 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:02:52.825938 392 external_mini_cluster.cc:1428] Setting key 6a44ec614b642cd01b0fb27052a9bcb5
I20260924 03:02:52.827582 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37809
--builtin_ntp_servers=127.0.98.20:32875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 03:02:52.827813 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.824145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42027 (local address 127.0.98.62:37809)
0924 03:02:52.824239 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:52.824242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:52.824553 (+ 311us) server_negotiation.cc:407] Connection header received
0924 03:02:52.825204 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:52.825206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:52.825213 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:52.825348 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:52.827441 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:52.827442 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:52.827442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:02:52.827447 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:52.827465 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:02:52.827470 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:52.827543 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:52.827696 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 03:02:52.827713 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 03:02:52.828060 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.824156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37809 (local address 127.0.98.2:42027)
0924 03:02:52.824401 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:52.824436 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:52.824937 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:52.825441 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:52.825447 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:52.825461 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:52.827269 (+ 1808us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:02:52.827270 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:52.827305 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:02:52.827343 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:52.827545 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:52.827548 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:52.827551 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:02:52.827708 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 03:02:52.827800 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":110,"threads_started":1}
I20260924 03:02:52.828577 5339 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37809
I20260924 03:02:52.828688 5339 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:52.828855 5339 heartbeater.cc:511] Master 127.0.98.62:37809 requested a full tablet report, sending...
I20260924 03:02:52.829268 5020 ts_manager.cc:194] Registered new tserver with Master: 122d701c49594c23aeadc91b695fa723 (127.0.98.2:41989)
W20260924 03:02:52.910615 5342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:52.910830 5342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:52.910851 5342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:52.912201 5342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:52.912257 5342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:52.912293 5342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:02:52.913708 5342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:52.913964 5342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:52.914273 5342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:52.919502 5347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:52.919478 5350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:52.919479 5348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:52.919816 5342 server_base.cc:1034] running on GCE node
I20260924 03:02:52.920063 5342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:52.920328 5342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:52.921561 5342 hybrid_clock.cc:648] HybridClock initialized: now 1790218972921541 us; error 34 us; skew 500 ppm
I20260924 03:02:52.923077 5342 webserver.cc:460] Webserver started at http://127.0.98.3:42015/ using document root <none> and password file <none>
I20260924 03:02:52.923329 5342 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:52.923400 5342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:52.923539 5342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:52.924973 5342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-2/data/instance:
uuid: "e41e8dbe5e95456d80c118469f86b6ad"
format_stamp: "Formatted at 2026-09-24 03:02:52 on dist-test-slave-wgsb"
server_key: "e11942da41b3df2b8828b46d978ed4aa"
server_key_iv: "6ffcab6fcb1ef1a07c2dd3a37750feca"
server_key_version: "encryptionkey@0"
I20260924 03:02:52.925482 5342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e41e8dbe5e95456d80c118469f86b6ad"
format_stamp: "Formatted at 2026-09-24 03:02:52 on dist-test-slave-wgsb"
server_key: "e11942da41b3df2b8828b46d978ed4aa"
server_key_iv: "6ffcab6fcb1ef1a07c2dd3a37750feca"
server_key_version: "encryptionkey@0"
I20260924 03:02:52.926807 5342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 03:02:52.927570 5356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:52.927892 5342 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:52.927991 5342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "e41e8dbe5e95456d80c118469f86b6ad"
format_stamp: "Formatted at 2026-09-24 03:02:52 on dist-test-slave-wgsb"
server_key: "e11942da41b3df2b8828b46d978ed4aa"
server_key_iv: "6ffcab6fcb1ef1a07c2dd3a37750feca"
server_key_version: "encryptionkey@0"
I20260924 03:02:52.928081 5342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:52.931439 5342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:52.931742 5342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:52.931885 5342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:52.932086 5342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:52.932415 5342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:52.932462 5342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:52.932494 5342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:52.932523 5342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:52.938258 5342 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39141
I20260924 03:02:52.938298 5469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39141 every 8 connection(s)
I20260924 03:02:52.938604 5342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:02:52.941443 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.939287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40271 (local address 127.0.98.62:37809)
0924 03:02:52.939337 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:52.939338 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:52.939574 (+ 236us) server_negotiation.cc:407] Connection header received
0924 03:02:52.940113 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:52.940114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:52.940121 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:52.940174 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:52.941232 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:52.941233 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:52.941233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:02:52.941238 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:52.941252 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:02:52.941256 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:52.941320 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:52.941376 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 03:02:52.941388 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:02:52.941612 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.939300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37809 (local address 127.0.98.3:40271)
0924 03:02:52.939444 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:52.939475 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:52.940023 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:52.940228 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:52.940231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:52.940239 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:52.941094 (+ 855us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:02:52.941095 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:52.941121 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:02:52.941147 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:52.941320 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:52.941323 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:52.941325 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:02:52.941375 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 03:02:52.941420 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":66,"threads_started":1}
I20260924 03:02:52.942140 5470 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37809
I20260924 03:02:52.942226 5470 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:52.942333 5470 heartbeater.cc:511] Master 127.0.98.62:37809 requested a full tablet report, sending...
I20260924 03:02:52.942667 5020 ts_manager.cc:194] Registered new tserver with Master: e41e8dbe5e95456d80c118469f86b6ad (127.0.98.3:39141)
I20260924 03:02:52.942708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 5342
I20260924 03:02:52.942836 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:02:52.943037 392 external_mini_cluster.cc:1428] Setting key cb3368f06b99f501a2029e47bda4fe80
I20260924 03:02:52.943887 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:02:52.946396 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.945201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37294 (local address 127.0.98.62:37809)
0924 03:02:52.945294 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:52.945297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:52.945407 (+ 110us) server_negotiation.cc:407] Connection header received
0924 03:02:52.945482 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:52.945482 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:52.945488 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:52.945532 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:52.946131 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:52.946132 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:52.946133 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:02:52.946137 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:52.946151 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:02:52.946155 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:52.946249 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:52.946309 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 03:02:52.946324 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 03:02:52.948352 5020 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:02:52.950028 5020 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:02:52.954022 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.952296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37393 (local address 127.0.0.1:51134)
0924 03:02:52.952754 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:52.952763 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:52.952806 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:52.952990 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:52.952995 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:52.953002 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:52.953537 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:02:52.953538 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:52.953555 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:02:52.953579 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:52.953835 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:52.953841 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:52.953844 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:02:52.953922 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 03:02:52.953937 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":3,"thread_start_us":90,"threads_started":1}
I20260924 03:02:52.954070 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.952388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51134 (local address 127.0.98.1:37393)
0924 03:02:52.952563 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:52.952568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:52.952871 (+ 303us) server_negotiation.cc:407] Connection header received
0924 03:02:52.952901 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:52.952904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:52.952914 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:52.952988 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:52.953683 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:52.953692 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:52.953693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:02:52.953698 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:52.953732 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:02:52.953757 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:52.953822 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:52.953950 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 03:02:52.953968 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":100,"threads_started":1}
I20260924 03:02:52.954192 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.952525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38332 (local address 127.0.98.3:39141)
0924 03:02:52.952699 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:52.952703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:52.952717 (+ 14us) server_negotiation.cc:407] Connection header received
0924 03:02:52.952795 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:52.952797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:52.952807 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:52.952879 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:52.953773 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:52.953780 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:52.953780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:02:52.953784 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:52.953818 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:02:52.953840 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:52.953892 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:52.954059 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 03:02:52.954076 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":122,"threads_started":1}
I20260924 03:02:52.954479 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.952579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39782 (local address 127.0.98.2:41989)
0924 03:02:52.953341 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:52.953345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:52.953354 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:02:52.953381 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:52.953384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:52.953392 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:52.953467 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:52.954047 (+ 580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:52.954053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:52.954053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:02:52.954058 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:52.954115 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:02:52.954149 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:52.954208 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:52.954408 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 03:02:52.954421 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":90,"threads_started":1}
I20260924 03:02:52.954362 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.952296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41989 (local address 127.0.0.1:39782)
0924 03:02:52.952830 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:52.952838 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:52.952869 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:52.953495 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:52.953496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:52.953498 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:52.953951 (+ 453us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:02:52.953951 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:52.953959 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:02:52.953969 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:52.954233 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:52.954233 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:52.954234 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 03:02:52.954280 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 03:02:52.954293 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":128,"thread_start_us":58,"threads_started":1}
I20260924 03:02:52.954022 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:52.952296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39141 (local address 127.0.0.1:38332)
0924 03:02:52.952631 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:52.952671 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:52.952730 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:52.953189 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:52.953189 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:52.953192 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:52.953678 (+ 486us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:02:52.953679 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:52.953683 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:02:52.953692 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:52.953890 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:52.953891 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:52.953891 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 03:02:52.953942 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 03:02:52.953952 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":64,"threads_started":1}
I20260924 03:02:52.955199 5273 tablet_service.cc:1467] Processing CreateTablet for tablet 07ff7ffd25394320957eff8b985de5d0 (DEFAULT_TABLE table=test-table [id=6a564bc0397a4f19b248ddb87605250e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:52.955199 5142 tablet_service.cc:1467] Processing CreateTablet for tablet 07ff7ffd25394320957eff8b985de5d0 (DEFAULT_TABLE table=test-table [id=6a564bc0397a4f19b248ddb87605250e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:52.955245 5404 tablet_service.cc:1467] Processing CreateTablet for tablet 07ff7ffd25394320957eff8b985de5d0 (DEFAULT_TABLE table=test-table [id=6a564bc0397a4f19b248ddb87605250e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:52.955504 5404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07ff7ffd25394320957eff8b985de5d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:52.955525 5273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07ff7ffd25394320957eff8b985de5d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:52.955525 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07ff7ffd25394320957eff8b985de5d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:52.958014 5484 tablet_bootstrap.cc:492] T 07ff7ffd25394320957eff8b985de5d0 P e41e8dbe5e95456d80c118469f86b6ad: Bootstrap starting.
I20260924 03:02:52.958627 5484 tablet_bootstrap.cc:654] T 07ff7ffd25394320957eff8b985de5d0 P e41e8dbe5e95456d80c118469f86b6ad: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:52.958732 5486 tablet_bootstrap.cc:492] T 07ff7ffd25394320957eff8b985de5d0 P dca1ff11088a46fe8e43b6b701377131: Bootstrap starting.
I20260924 03:02:52.959148 5484 log.cc:824] T 07ff7ffd25394320957eff8b985de5d0 P e41e8dbe5e95456d80c118469f86b6ad: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:52.959398 5486 tablet_bootstrap.cc:654] T 07ff7ffd25394320957eff8b985de5d0 P dca1ff11088a46fe8e43b6b701377131: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:52.959525 5485 tablet_bootstrap.cc:492] T 07ff7ffd25394320957eff8b985de5d0 P 122d701c49594c23aeadc91b695fa723: Bootstrap starting.
I20260924 03:02:52.959771 5486 log.cc:824] T 07ff7ffd25394320957eff8b985de5d0 P dca1ff11088a46fe8e43b6b701377131: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:52.960001 5484 tablet_bootstrap.cc:492] T 07ff7ffd25394320957eff8b985de5d0 P e41e8dbe5e95456d80c118469f86b6ad: No bootstrap required, opened a new log
I20260924 03:02:52.960083 5484 ts_tablet_manager.cc:1397] T 07ff7ffd25394320957eff8b985de5d0 P e41e8dbe5e95456d80c118469f86b6ad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:02:52.960208 5485 tablet_bootstrap.cc:654] T 07ff7ffd25394320957eff8b985de5d0 P 122d701c49594c23aeadc91b695fa723: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:52.960520 5485 log.cc:824] T 07ff7ffd25394320957eff8b985de5d0 P 122d701c49594c23aeadc91b695fa723: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:52.960551 5486 tablet_bootstrap.cc:492] T 07ff7ffd25394320957eff8b985de5d0 P dca1ff11088a46fe8e43b6b701377131: No bootstrap required, opened a new log
I20260924 03:02:52.960604 5486 ts_tablet_manager.cc:1397] T 07ff7ffd25394320957eff8b985de5d0 P dca1ff11088a46fe8e43b6b701377131: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 03:02:52.961304 5485 tablet_bootstrap.cc:492] T 07ff7ffd25394320957eff8b985de5d0 P 122d701c49594c23aeadc91b695fa723: No bootstrap required, opened a new log
I20260924 03:02:52.961374 5485 ts_tablet_manager.cc:1397] T 07ff7ffd25394320957eff8b985de5d0 P 122d701c49594c23aeadc91b695fa723: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 03:02:52.962473 5484 raft_consensus.cc:348] T 07ff7ffd25394320957eff8b985de5d0 P e41e8dbe5e95456d80c118469f86b6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41e8dbe5e95456d80c118469f86b6ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39141 } } peers { permanent_uuid: "dca1ff11088a46fe8e43b6b701377131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37393 } } peers { permanent_uuid: "122d701c49594c23aeadc91b695fa723" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41989 } }
I20260924 03:02:52.962694 5484 raft_consensus.cc:374] T 07ff7ffd25394320957eff8b985de5d0 P e41e8dbe5e95456d80c118469f86b6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:52.962740 5484 raft_consensus.cc:729] T 07ff7ffd25394320957eff8b985de5d0 P e41e8dbe5e95456d80c118469f86b6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41e8dbe5e95456d80c118469f86b6ad, State: Initialized, Role: FOLLOWER
I20260924 03:02:52.962849 5484 consensus_queue.cc:260] T 07ff7ffd25394320957eff8b985de5d0 P e41e8dbe5e95456d80c118469f86b6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41e8dbe5e95456d80c118469f86b6ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39141 } } peers { permanent_uuid: "dca1ff11088a46fe8e43b6b701377131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37393 } } peers { permanent_uuid: "122d701c49594c23aeadc91b695fa723" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41989 } }
I20260924 03:02:52.963152 5484 ts_tablet_manager.cc:1428] T 07ff7ffd25394320957eff8b985de5d0 P e41e8dbe5e95456d80c118469f86b6ad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:02:52.963296 5470 heartbeater.cc:503] Master 127.0.98.62:37809 was elected leader, sending a full tablet report...
I20260924 03:02:52.963196 5486 raft_consensus.cc:348] T 07ff7ffd25394320957eff8b985de5d0 P dca1ff11088a46fe8e43b6b701377131 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41e8dbe5e95456d80c118469f86b6ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39141 } } peers { permanent_uuid: "dca1ff11088a46fe8e43b6b701377131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37393 } } peers { permanent_uuid: "122d701c49594c23aeadc91b695fa723" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41989 } }
I20260924 03:02:52.963351 5486 raft_consensus.cc:374] T 07ff7ffd25394320957eff8b985de5d0 P dca1ff11088a46fe8e43b6b701377131 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:52.963383 5486 raft_consensus.cc:729] T 07ff7ffd25394320957eff8b985de5d0 P dca1ff11088a46fe8e43b6b701377131 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca1ff11088a46fe8e43b6b701377131, State: Initialized, Role: FOLLOWER
I20260924 03:02:52.963495 5486 consensus_queue.cc:260] T 07ff7ffd25394320957eff8b985de5d0 P dca1ff11088a46fe8e43b6b701377131 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41e8dbe5e95456d80c118469f86b6ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39141 } } peers { permanent_uuid: "dca1ff11088a46fe8e43b6b701377131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37393 } } peers { permanent_uuid: "122d701c49594c23aeadc91b695fa723" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41989 } }
I20260924 03:02:52.963804 5486 ts_tablet_manager.cc:1428] T 07ff7ffd25394320957eff8b985de5d0 P dca1ff11088a46fe8e43b6b701377131: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 03:02:52.963930 5208 heartbeater.cc:503] Master 127.0.98.62:37809 was elected leader, sending a full tablet report...
I20260924 03:02:52.964887 5485 raft_consensus.cc:348] T 07ff7ffd25394320957eff8b985de5d0 P 122d701c49594c23aeadc91b695fa723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41e8dbe5e95456d80c118469f86b6ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39141 } } peers { permanent_uuid: "dca1ff11088a46fe8e43b6b701377131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37393 } } peers { permanent_uuid: "122d701c49594c23aeadc91b695fa723" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41989 } }
I20260924 03:02:52.965003 5485 raft_consensus.cc:374] T 07ff7ffd25394320957eff8b985de5d0 P 122d701c49594c23aeadc91b695fa723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:52.965022 5485 raft_consensus.cc:729] T 07ff7ffd25394320957eff8b985de5d0 P 122d701c49594c23aeadc91b695fa723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 122d701c49594c23aeadc91b695fa723, State: Initialized, Role: FOLLOWER
I20260924 03:02:52.965126 5485 consensus_queue.cc:260] T 07ff7ffd25394320957eff8b985de5d0 P 122d701c49594c23aeadc91b695fa723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41e8dbe5e95456d80c118469f86b6ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39141 } } peers { permanent_uuid: "dca1ff11088a46fe8e43b6b701377131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37393 } } peers { permanent_uuid: "122d701c49594c23aeadc91b695fa723" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41989 } }
I20260924 03:02:52.965341 5485 ts_tablet_manager.cc:1428] T 07ff7ffd25394320957eff8b985de5d0 P 122d701c49594c23aeadc91b695fa723: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260924 03:02:52.965409 5339 heartbeater.cc:503] Master 127.0.98.62:37809 was elected leader, sending a full tablet report...
I20260924 03:02:53.024179 5491 raft_consensus.cc:482] T 07ff7ffd25394320957eff8b985de5d0 P dca1ff11088a46fe8e43b6b701377131 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:02:53.024312 5491 raft_consensus.cc:504] T 07ff7ffd25394320957eff8b985de5d0 P dca1ff11088a46fe8e43b6b701377131 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41e8dbe5e95456d80c118469f86b6ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39141 } } peers { permanent_uuid: "dca1ff11088a46fe8e43b6b701377131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37393 } } peers { permanent_uuid: "122d701c49594c23aeadc91b695fa723" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41989 } }
I20260924 03:02:53.024657 5491 leader_election.cc:288] T 07ff7ffd25394320957eff8b985de5d0 P dca1ff11088a46fe8e43b6b701377131 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e41e8dbe5e95456d80c118469f86b6ad (127.0.98.3:39141), 122d701c49594c23aeadc91b695fa723 (127.0.98.2:41989)
I20260924 03:02:53.026427 5210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:53.024955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41989 (local address 127.0.98.1:46911)
0924 03:02:53.025027 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:53.025041 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:53.025085 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:53.025342 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:53.025343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:53.025346 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:53.025849 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:02:53.025850 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:53.025859 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:02:53.025875 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:53.026240 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:53.026241 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:53.026243 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:02:53.026322 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 03:02:53.026338 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 03:02:53.026500 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:53.025016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46911 (local address 127.0.98.2:41989)
0924 03:02:53.025066 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:53.025068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:53.025080 (+ 12us) server_negotiation.cc:407] Connection header received
0924 03:02:53.025183 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:53.025184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:53.025189 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:53.025321 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:53.026058 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:53.026060 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:53.026060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:02:53.026066 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:53.026088 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:02:53.026093 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:53.026176 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:53.026394 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 03:02:53.026415 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 03:02:53.026562 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:53.025044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60585 (local address 127.0.98.3:39141)
0924 03:02:53.025084 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:53.025086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:53.025300 (+ 214us) server_negotiation.cc:407] Connection header received
0924 03:02:53.025363 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:53.025364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:53.025372 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:53.025425 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:53.026249 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:53.026282 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:53.026283 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:02:53.026289 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:53.026308 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:02:53.026314 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:53.026373 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:53.026469 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 03:02:53.026482 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:02:53.026854 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:53.024951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39141 (local address 127.0.98.1:60585)
0924 03:02:53.025206 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:53.025239 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:53.025305 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:53.025448 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:53.025449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:53.025452 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:53.026087 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:02:53.026087 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:53.026094 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:02:53.026108 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:53.026397 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:53.026399 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:53.026399 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 03:02:53.026469 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 03:02:53.026482 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":14,"thread_start_us":126,"threads_started":1}
I20260924 03:02:53.027299 5424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ff7ffd25394320957eff8b985de5d0" candidate_uuid: "dca1ff11088a46fe8e43b6b701377131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41e8dbe5e95456d80c118469f86b6ad" is_pre_election: true
I20260924 03:02:53.027288 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ff7ffd25394320957eff8b985de5d0" candidate_uuid: "dca1ff11088a46fe8e43b6b701377131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122d701c49594c23aeadc91b695fa723" is_pre_election: true
I20260924 03:02:53.027628 5424 raft_consensus.cc:2454] T 07ff7ffd25394320957eff8b985de5d0 P e41e8dbe5e95456d80c118469f86b6ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca1ff11088a46fe8e43b6b701377131 in term 0.
I20260924 03:02:53.027637 5293 raft_consensus.cc:2454] T 07ff7ffd25394320957eff8b985de5d0 P 122d701c49594c23aeadc91b695fa723 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca1ff11088a46fe8e43b6b701377131 in term 0.
I20260924 03:02:53.028394 5095 leader_election.cc:302] T 07ff7ffd25394320957eff8b985de5d0 P dca1ff11088a46fe8e43b6b701377131 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dca1ff11088a46fe8e43b6b701377131, e41e8dbe5e95456d80c118469f86b6ad; no voters:
I20260924 03:02:53.028855 5491 raft_consensus.cc:2781] T 07ff7ffd25394320957eff8b985de5d0 P dca1ff11088a46fe8e43b6b701377131 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:02:53.028919 5491 raft_consensus.cc:482] T 07ff7ffd25394320957eff8b985de5d0 P dca1ff11088a46fe8e43b6b701377131 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:02:53.028937 5491 raft_consensus.cc:3037] T 07ff7ffd25394320957eff8b985de5d0 P dca1ff11088a46fe8e43b6b701377131 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:53.029685 5491 raft_consensus.cc:504] T 07ff7ffd25394320957eff8b985de5d0 P dca1ff11088a46fe8e43b6b701377131 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41e8dbe5e95456d80c118469f86b6ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39141 } } peers { permanent_uuid: "dca1ff11088a46fe8e43b6b701377131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37393 } } peers { permanent_uuid: "122d701c49594c23aeadc91b695fa723" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41989 } }
I20260924 03:02:53.029855 5491 leader_election.cc:288] T 07ff7ffd25394320957eff8b985de5d0 P dca1ff11088a46fe8e43b6b701377131 [CANDIDATE]: Term 1 election: Requested vote from peers e41e8dbe5e95456d80c118469f86b6ad (127.0.98.3:39141), 122d701c49594c23aeadc91b695fa723 (127.0.98.2:41989)
I20260924 03:02:53.030025 5424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ff7ffd25394320957eff8b985de5d0" candidate_uuid: "dca1ff11088a46fe8e43b6b701377131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41e8dbe5e95456d80c118469f86b6ad"
I20260924 03:02:53.030078 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ff7ffd25394320957eff8b985de5d0" candidate_uuid: "dca1ff11088a46fe8e43b6b701377131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122d701c49594c23aeadc91b695fa723"
I20260924 03:02:53.030144 5424 raft_consensus.cc:3037] T 07ff7ffd25394320957eff8b985de5d0 P e41e8dbe5e95456d80c118469f86b6ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:53.030165 5293 raft_consensus.cc:3037] T 07ff7ffd25394320957eff8b985de5d0 P 122d701c49594c23aeadc91b695fa723 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:53.031035 5293 raft_consensus.cc:2454] T 07ff7ffd25394320957eff8b985de5d0 P 122d701c49594c23aeadc91b695fa723 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca1ff11088a46fe8e43b6b701377131 in term 1.
I20260924 03:02:53.031035 5424 raft_consensus.cc:2454] T 07ff7ffd25394320957eff8b985de5d0 P e41e8dbe5e95456d80c118469f86b6ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca1ff11088a46fe8e43b6b701377131 in term 1.
I20260924 03:02:53.031203 5098 leader_election.cc:302] T 07ff7ffd25394320957eff8b985de5d0 P dca1ff11088a46fe8e43b6b701377131 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 122d701c49594c23aeadc91b695fa723, dca1ff11088a46fe8e43b6b701377131; no voters:
I20260924 03:02:53.031313 5491 raft_consensus.cc:2781] T 07ff7ffd25394320957eff8b985de5d0 P dca1ff11088a46fe8e43b6b701377131 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:53.031545 5491 raft_consensus.cc:686] T 07ff7ffd25394320957eff8b985de5d0 P dca1ff11088a46fe8e43b6b701377131 [term 1 LEADER]: Becoming Leader. State: Replica: dca1ff11088a46fe8e43b6b701377131, State: Running, Role: LEADER
I20260924 03:02:53.031689 5491 consensus_queue.cc:237] T 07ff7ffd25394320957eff8b985de5d0 P dca1ff11088a46fe8e43b6b701377131 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41e8dbe5e95456d80c118469f86b6ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39141 } } peers { permanent_uuid: "dca1ff11088a46fe8e43b6b701377131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37393 } } peers { permanent_uuid: "122d701c49594c23aeadc91b695fa723" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41989 } }
I20260924 03:02:53.032577 5019 catalog_manager.cc:5495] T 07ff7ffd25394320957eff8b985de5d0 P dca1ff11088a46fe8e43b6b701377131 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca1ff11088a46fe8e43b6b701377131 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "dca1ff11088a46fe8e43b6b701377131" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41e8dbe5e95456d80c118469f86b6ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca1ff11088a46fe8e43b6b701377131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "122d701c49594c23aeadc91b695fa723" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41989 } health_report { overall_health: UNKNOWN } } }
I20260924 03:02:53.051570 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:53.050567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51144 (local address 127.0.98.1:37393)
0924 03:02:53.050635 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:53.050637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:53.050646 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:02:53.050689 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:53.050690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:53.050707 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:53.050765 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:53.051324 (+ 559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:53.051325 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:53.051325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:02:53.051330 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:53.051346 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:02:53.051350 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:53.051422 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:53.051478 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 03:02:53.051493 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 03:02:53.053344 5293 raft_consensus.cc:1261] T 07ff7ffd25394320957eff8b985de5d0 P 122d701c49594c23aeadc91b695fa723 [term 1 FOLLOWER]: Refusing update from remote peer dca1ff11088a46fe8e43b6b701377131: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:02:53.053408 5424 raft_consensus.cc:1261] T 07ff7ffd25394320957eff8b985de5d0 P e41e8dbe5e95456d80c118469f86b6ad [term 1 FOLLOWER]: Refusing update from remote peer dca1ff11088a46fe8e43b6b701377131: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:02:53.053692 5491 consensus_queue.cc:1035] T 07ff7ffd25394320957eff8b985de5d0 P dca1ff11088a46fe8e43b6b701377131 [LEADER]: Connected to new peer: Peer: permanent_uuid: "122d701c49594c23aeadc91b695fa723" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:02:53.053833 5494 consensus_queue.cc:1035] T 07ff7ffd25394320957eff8b985de5d0 P dca1ff11088a46fe8e43b6b701377131 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41e8dbe5e95456d80c118469f86b6ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:02:53.056159 5497 mvcc.cc:205] Tried to move back new op lower bound from 7332736913623932928 to 7332736913538359296. Current Snapshot: MvccSnapshot[applied={T|T < 7332736913623932928}]
I20260924 03:02:53.056737 5498 mvcc.cc:205] Tried to move back new op lower bound from 7332736913623932928 to 7332736913538359296. Current Snapshot: MvccSnapshot[applied={T|T < 7332736913623932928}]
I20260924 03:02:53.058962 5020 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:02:53.059060 5020 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:02:53.059813 5020 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f19bb3814ea40c58a9daffbb2719881: Sending DeleteTablet for 3 replicas of tablet 07ff7ffd25394320957eff8b985de5d0
I20260924 03:02:53.060140 5142 tablet_service.cc:1514] Processing DeleteTablet for tablet 07ff7ffd25394320957eff8b985de5d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:02:53 UTC) from {username='slave'} at 127.0.0.1:51134
I20260924 03:02:53.060153 5273 tablet_service.cc:1514] Processing DeleteTablet for tablet 07ff7ffd25394320957eff8b985de5d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:02:53 UTC) from {username='slave'} at 127.0.0.1:39782
I20260924 03:02:53.060392 5508 tablet_replica.cc:330] T 07ff7ffd25394320957eff8b985de5d0 P 122d701c49594c23aeadc91b695fa723: stopping tablet replica
I20260924 03:02:53.060386 5404 tablet_service.cc:1514] Processing DeleteTablet for tablet 07ff7ffd25394320957eff8b985de5d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:02:53 UTC) from {username='slave'} at 127.0.0.1:38332
I20260924 03:02:53.060468 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 5080
I20260924 03:02:53.060492 5508 raft_consensus.cc:2229] T 07ff7ffd25394320957eff8b985de5d0 P 122d701c49594c23aeadc91b695fa723 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:02:53.060590 5508 raft_consensus.cc:2258] T 07ff7ffd25394320957eff8b985de5d0 P 122d701c49594c23aeadc91b695fa723 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:02:53.061328 5508 ts_tablet_manager.cc:1905] T 07ff7ffd25394320957eff8b985de5d0 P 122d701c49594c23aeadc91b695fa723: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:02:53.063262 5508 ts_tablet_manager.cc:1918] T 07ff7ffd25394320957eff8b985de5d0 P 122d701c49594c23aeadc91b695fa723: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:02:53.063349 5508 log.cc:1196] T 07ff7ffd25394320957eff8b985de5d0 P 122d701c49594c23aeadc91b695fa723: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-1/wal/wals/07ff7ffd25394320957eff8b985de5d0
I20260924 03:02:53.063627 5508 ts_tablet_manager.cc:1939] T 07ff7ffd25394320957eff8b985de5d0 P 122d701c49594c23aeadc91b695fa723: Deleting consensus metadata
I20260924 03:02:53.064049 5007 catalog_manager.cc:4841] TS 122d701c49594c23aeadc91b695fa723 (127.0.98.2:41989): tablet 07ff7ffd25394320957eff8b985de5d0 (table test-table [id=6a564bc0397a4f19b248ddb87605250e]) successfully deleted
W20260924 03:02:53.065742 5005 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37393 (error 108)
I20260924 03:02:53.066020 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 5211
I20260924 03:02:53.066035 5509 tablet_replica.cc:330] T 07ff7ffd25394320957eff8b985de5d0 P e41e8dbe5e95456d80c118469f86b6ad: stopping tablet replica
I20260924 03:02:53.066531 5509 raft_consensus.cc:2229] T 07ff7ffd25394320957eff8b985de5d0 P e41e8dbe5e95456d80c118469f86b6ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:02:53.066584 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:53.066086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37393 (local address 127.0.0.1:51158)
0924 03:02:53.066509 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:53.066550 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404}
I20260924 03:02:53.066658 5509 raft_consensus.cc:2258] T 07ff7ffd25394320957eff8b985de5d0 P e41e8dbe5e95456d80c118469f86b6ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:02:53.067436 5509 ts_tablet_manager.cc:1905] T 07ff7ffd25394320957eff8b985de5d0 P e41e8dbe5e95456d80c118469f86b6ad: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 03:02:53.067960 5005 catalog_manager.cc:4584] TS dca1ff11088a46fe8e43b6b701377131 (127.0.98.1:37393): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07ff7ffd25394320957eff8b985de5d0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37393: connect: Connection refused (error 111)
I20260924 03:02:53.069432 5509 ts_tablet_manager.cc:1918] T 07ff7ffd25394320957eff8b985de5d0 P e41e8dbe5e95456d80c118469f86b6ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:02:53.069507 5509 log.cc:1196] T 07ff7ffd25394320957eff8b985de5d0 P e41e8dbe5e95456d80c118469f86b6ad: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218948286999-392-0/minicluster-data/ts-2/wal/wals/07ff7ffd25394320957eff8b985de5d0
I20260924 03:02:53.069751 5509 ts_tablet_manager.cc:1939] T 07ff7ffd25394320957eff8b985de5d0 P e41e8dbe5e95456d80c118469f86b6ad: Deleting consensus metadata
I20260924 03:02:53.070210 5004 catalog_manager.cc:4841] TS e41e8dbe5e95456d80c118469f86b6ad (127.0.98.3:39141): tablet 07ff7ffd25394320957eff8b985de5d0 (table test-table [id=6a564bc0397a4f19b248ddb87605250e]) successfully deleted
I20260924 03:02:53.071723 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 5342
I20260924 03:02:53.076658 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 4989
2026-09-24T03:02:53Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (660 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 03:02:53.095942 392 mini_oidc.cc:152] Starting JWKS server
I20260924 03:02:53.096975 392 webserver.cc:460] Webserver started at https://127.0.0.1:32843/ using document root <none> and password file <none>
I20260924 03:02:53.097093 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 03:02:53.097277 392 webserver.cc:460] Webserver started at http://127.0.0.1:41095/ using document root <none> and password file <none>
2026-09-24T03:02:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:02:53Z Disabled control of system clock
I20260924 03:02:53.105141 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41441
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41441
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32843/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218948286999-392-0/testchainca.cert with env {}
W20260924 03:02:53.188889 5516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:53.189087 5516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:53.189109 5516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:53.190276 5516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:02:53.190317 5516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:53.190330 5516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:53.190342 5516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:02:53.190356 5516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 03:02:53.190367 5516 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 03:02:53.190379 5516 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32843/jwks.json
I20260924 03:02:53.191881 5516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41441
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32843/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218948286999-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:53.192085 5516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:53.192255 5516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:02:53.197481 5516 server_base.cc:1034] running on GCE node
W20260924 03:02:53.197465 5524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:53.197465 5521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:53.197455 5522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:53.197962 5516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:53.198268 5516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:53.199431 5516 hybrid_clock.cc:648] HybridClock initialized: now 1790218973199412 us; error 39 us; skew 500 ppm
I20260924 03:02:53.200729 5516 webserver.cc:460] Webserver started at http://127.0.98.62:36201/ using document root <none> and password file <none>
I20260924 03:02:53.200987 5516 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:53.201037 5516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:53.201206 5516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:53.202237 5516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "0c12e44f37ac46f286c48931718f4422"
format_stamp: "Formatted at 2026-09-24 03:02:53 on dist-test-slave-wgsb"
server_key: "b6d652287cfbd1cb77f0242731588b33"
server_key_iv: "2a5c68a91aa1e3f423154ff4d69c216f"
server_key_version: "encryptionkey@0"
I20260924 03:02:53.202602 5516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "0c12e44f37ac46f286c48931718f4422"
format_stamp: "Formatted at 2026-09-24 03:02:53 on dist-test-slave-wgsb"
server_key: "b6d652287cfbd1cb77f0242731588b33"
server_key_iv: "2a5c68a91aa1e3f423154ff4d69c216f"
server_key_version: "encryptionkey@0"
I20260924 03:02:53.203954 5516 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 03:02:53.204722 5530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:53.204905 5516 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:02:53.205008 5516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "0c12e44f37ac46f286c48931718f4422"
format_stamp: "Formatted at 2026-09-24 03:02:53 on dist-test-slave-wgsb"
server_key: "b6d652287cfbd1cb77f0242731588b33"
server_key_iv: "2a5c68a91aa1e3f423154ff4d69c216f"
server_key_version: "encryptionkey@0"
I20260924 03:02:53.205082 5516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:53.227059 5516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:53.227387 5516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:53.227535 5516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:53.231374 5516 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41441
I20260924 03:02:53.231410 5582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41441 every 8 connection(s)
I20260924 03:02:53.231812 5516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:02:53.232808 5583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:53.235055 5583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c12e44f37ac46f286c48931718f4422: Bootstrap starting.
I20260924 03:02:53.235718 5583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c12e44f37ac46f286c48931718f4422: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:53.236013 5583 log.cc:824] T 00000000000000000000000000000000 P 0c12e44f37ac46f286c48931718f4422: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:53.236699 5583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c12e44f37ac46f286c48931718f4422: No bootstrap required, opened a new log
I20260924 03:02:53.238265 5583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c12e44f37ac46f286c48931718f4422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c12e44f37ac46f286c48931718f4422" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41441 } }
I20260924 03:02:53.238401 5583 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c12e44f37ac46f286c48931718f4422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:53.238461 5583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c12e44f37ac46f286c48931718f4422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c12e44f37ac46f286c48931718f4422, State: Initialized, Role: FOLLOWER
I20260924 03:02:53.238619 5583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c12e44f37ac46f286c48931718f4422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c12e44f37ac46f286c48931718f4422" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41441 } }
I20260924 03:02:53.238689 5583 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c12e44f37ac46f286c48931718f4422 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:02:53.238710 5583 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c12e44f37ac46f286c48931718f4422 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:02:53.238739 5583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c12e44f37ac46f286c48931718f4422 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:53.239269 5583 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c12e44f37ac46f286c48931718f4422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c12e44f37ac46f286c48931718f4422" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41441 } }
I20260924 03:02:53.239377 5583 leader_election.cc:302] T 00000000000000000000000000000000 P 0c12e44f37ac46f286c48931718f4422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c12e44f37ac46f286c48931718f4422; no voters:
I20260924 03:02:53.239539 5583 leader_election.cc:288] T 00000000000000000000000000000000 P 0c12e44f37ac46f286c48931718f4422 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:02:53.239544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 5516
I20260924 03:02:53.239652 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:02:53.239760 5586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c12e44f37ac46f286c48931718f4422 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:53.239835 392 external_mini_cluster.cc:1428] Setting key 9cfc780256d1fbe15dda0e0d1b72a119
I20260924 03:02:53.239852 5583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c12e44f37ac46f286c48931718f4422 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:02:53.239938 5586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c12e44f37ac46f286c48931718f4422 [term 1 LEADER]: Becoming Leader. State: Replica: 0c12e44f37ac46f286c48931718f4422, State: Running, Role: LEADER
I20260924 03:02:53.240048 5586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c12e44f37ac46f286c48931718f4422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c12e44f37ac46f286c48931718f4422" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41441 } }
I20260924 03:02:53.240362 5586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c12e44f37ac46f286c48931718f4422 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c12e44f37ac46f286c48931718f4422. Latest consensus state: current_term: 1 leader_uuid: "0c12e44f37ac46f286c48931718f4422" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c12e44f37ac46f286c48931718f4422" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41441 } } }
I20260924 03:02:53.240449 5586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c12e44f37ac46f286c48931718f4422 [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:53.240566 5588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c12e44f37ac46f286c48931718f4422 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c12e44f37ac46f286c48931718f4422" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c12e44f37ac46f286c48931718f4422" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41441 } } }
I20260924 03:02:53.240687 5588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c12e44f37ac46f286c48931718f4422 [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:53.241266 5597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:02:53.241839 5597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:02:53.244225 5597 catalog_manager.cc:1347] Generated new cluster ID: 4729d64bb3414197b740d3a7559ee35a
I20260924 03:02:53.244283 5597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:02:53.245803 5595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:53.240429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59414 (local address 127.0.98.62:41441)
0924 03:02:53.240642 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:53.240652 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:53.240827 (+ 175us) server_negotiation.cc:407] Connection header received
0924 03:02:53.240953 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:53.240960 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:53.241035 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:53.241232 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:53.241464 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:53.242790 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:53.243620 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:53.244236 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:53.244365 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:53.245107 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:53.245111 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:53.245111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:02:53.245319 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:53.245357 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:02:53.245370 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:53.245433 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:53.245531 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 03:02:53.245604 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260924 03:02:53.246603 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:02:53.265230 5597 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:02:53.265973 5597 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:02:53.274732 5597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c12e44f37ac46f286c48931718f4422: Generated new TSK 0
I20260924 03:02:53.274941 5597 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 03:02:53.303296 5613 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 03:02:53.303418 5613 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 03:02:53.303658 5595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:53.286465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59420 (local address 127.0.98.62:41441)
0924 03:02:53.286551 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:53.286554 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:53.286707 (+ 153us) server_negotiation.cc:407] Connection header received
0924 03:02:53.286789 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:53.286791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:53.286813 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:53.286895 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0924 03:02:53.287073 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:53.287869 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:53.288690 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:53.288995 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:53.289064 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:53.289263 (+ 199us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 03:02:53.303404 (+ 14141us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 03:02:53.303554 (+ 150us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59420: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 03:02:53.303719 5595 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59420: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 03:02:53.304283 5612 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:02:53.286349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41441 (local address 127.0.0.1:59420)
0924 03:02:53.286623 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:53.286642 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:53.286708 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:53.286940 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:53.286941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:53.286943 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 03:02:53.286999 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:53.287001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:53.287979 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:53.287980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:53.288604 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:53.288606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:53.289071 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:53.289075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:53.289131 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:53.289148 (+ 17us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 03:02:53.303690 (+ 14542us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 03:02:53.303913 (+ 223us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41441: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":134,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:41441: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 03:02:53.304945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 5516
2026-09-24T03:02:53Z chronyd exiting
I20260924 03:02:53.341394 392 test_util.cc:175] -----------------------------------------------
I20260924 03:02:53.341508 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218948286999-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (246 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 03:02:53.342368 392 mini_oidc.cc:152] Starting JWKS server
I20260924 03:02:53.343108 392 webserver.cc:460] Webserver started at https://127.0.0.1:34397/ using document root <none> and password file <none>
I20260924 03:02:53.343179 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 03:02:53.343357 392 webserver.cc:460] Webserver started at http://127.0.0.1:41145/ using document root <none> and password file <none>
2026-09-24T03:02:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:02:53Z Disabled control of system clock
I20260924 03:02:53.351323 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42193
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42193
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34397/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218948286999-392-0/testchainca.cert with env {}
W20260924 03:02:53.431483 5619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:53.431706 5619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:53.431730 5619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:53.432927 5619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:02:53.432971 5619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:53.432983 5619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:53.432997 5619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:02:53.433039 5619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 03:02:53.433070 5619 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 03:02:53.433085 5619 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34397/jwks.json
I20260924 03:02:53.434638 5619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42193
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34397/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218948286999-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:53.434904 5619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:53.435127 5619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:53.439953 5624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:53.439922 5627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:53.439906 5625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:53.440245 5619 server_base.cc:1034] running on GCE node
I20260924 03:02:53.440495 5619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:53.440811 5619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:53.441982 5619 hybrid_clock.cc:648] HybridClock initialized: now 1790218973441975 us; error 51 us; skew 500 ppm
I20260924 03:02:53.443320 5619 webserver.cc:460] Webserver started at http://127.0.98.62:33623/ using document root <none> and password file <none>
I20260924 03:02:53.443576 5619 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:53.443747 5619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:53.443905 5619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:53.445051 5619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "8dc118972aca44f4901844e9c877147f"
format_stamp: "Formatted at 2026-09-24 03:02:53 on dist-test-slave-wgsb"
server_key: "62efefbb1187a56aed47db441350a1c2"
server_key_iv: "5a383f6ffbb29ede45007c01d5ec9264"
server_key_version: "encryptionkey@0"
I20260924 03:02:53.445438 5619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "8dc118972aca44f4901844e9c877147f"
format_stamp: "Formatted at 2026-09-24 03:02:53 on dist-test-slave-wgsb"
server_key: "62efefbb1187a56aed47db441350a1c2"
server_key_iv: "5a383f6ffbb29ede45007c01d5ec9264"
server_key_version: "encryptionkey@0"
I20260924 03:02:53.446723 5619 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:02:53.447482 5633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:53.447703 5619 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:53.447808 5619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "8dc118972aca44f4901844e9c877147f"
format_stamp: "Formatted at 2026-09-24 03:02:53 on dist-test-slave-wgsb"
server_key: "62efefbb1187a56aed47db441350a1c2"
server_key_iv: "5a383f6ffbb29ede45007c01d5ec9264"
server_key_version: "encryptionkey@0"
I20260924 03:02:53.447881 5619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:53.471624 5619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:53.471951 5619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:53.472111 5619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:53.476078 5619 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42193
I20260924 03:02:53.476111 5685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42193 every 8 connection(s)
I20260924 03:02:53.476455 5619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:02:53.477421 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:53.479801 5686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dc118972aca44f4901844e9c877147f: Bootstrap starting.
I20260924 03:02:53.480391 5686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dc118972aca44f4901844e9c877147f: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:53.480674 5686 log.cc:824] T 00000000000000000000000000000000 P 8dc118972aca44f4901844e9c877147f: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:53.481459 5686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dc118972aca44f4901844e9c877147f: No bootstrap required, opened a new log
I20260924 03:02:53.483021 5686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8dc118972aca44f4901844e9c877147f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc118972aca44f4901844e9c877147f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42193 } }
I20260924 03:02:53.483165 5686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8dc118972aca44f4901844e9c877147f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:53.483187 5686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8dc118972aca44f4901844e9c877147f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc118972aca44f4901844e9c877147f, State: Initialized, Role: FOLLOWER
I20260924 03:02:53.483374 5686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dc118972aca44f4901844e9c877147f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc118972aca44f4901844e9c877147f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42193 } }
I20260924 03:02:53.483457 5686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8dc118972aca44f4901844e9c877147f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:02:53.483496 5686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8dc118972aca44f4901844e9c877147f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:02:53.483546 5686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8dc118972aca44f4901844e9c877147f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:53.484095 5686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8dc118972aca44f4901844e9c877147f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc118972aca44f4901844e9c877147f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42193 } }
I20260924 03:02:53.484210 5686 leader_election.cc:302] T 00000000000000000000000000000000 P 8dc118972aca44f4901844e9c877147f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8dc118972aca44f4901844e9c877147f; no voters:
I20260924 03:02:53.484387 5686 leader_election.cc:288] T 00000000000000000000000000000000 P 8dc118972aca44f4901844e9c877147f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:02:53.484447 5689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8dc118972aca44f4901844e9c877147f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:53.484652 5686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8dc118972aca44f4901844e9c877147f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:02:53.484653 5689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8dc118972aca44f4901844e9c877147f [term 1 LEADER]: Becoming Leader. State: Replica: 8dc118972aca44f4901844e9c877147f, State: Running, Role: LEADER
I20260924 03:02:53.484793 5689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8dc118972aca44f4901844e9c877147f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc118972aca44f4901844e9c877147f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42193 } }
I20260924 03:02:53.485183 5691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dc118972aca44f4901844e9c877147f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dc118972aca44f4901844e9c877147f. Latest consensus state: current_term: 1 leader_uuid: "8dc118972aca44f4901844e9c877147f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc118972aca44f4901844e9c877147f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42193 } } }
I20260924 03:02:53.485354 5691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dc118972aca44f4901844e9c877147f [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:53.485342 5690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dc118972aca44f4901844e9c877147f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8dc118972aca44f4901844e9c877147f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc118972aca44f4901844e9c877147f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42193 } } }
I20260924 03:02:53.485736 5690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dc118972aca44f4901844e9c877147f [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:53.485733 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 5619
I20260924 03:02:53.485738 5698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:02:53.485834 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:02:53.486016 392 external_mini_cluster.cc:1428] Setting key 48c5c5913bad8f40c76df16e397a8be8
I20260924 03:02:53.486286 5698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:02:53.489114 5698 catalog_manager.cc:1347] Generated new cluster ID: 9a783c39d9f34f02be02bdba481ad6be
I20260924 03:02:53.489174 5698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:02:53.491839 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:53.486487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53080 (local address 127.0.98.62:42193)
0924 03:02:53.486907 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:53.486916 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:53.486945 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:02:53.487048 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:53.487053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:53.487106 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:53.487254 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:53.487569 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:53.488966 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:53.489861 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:53.490286 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:53.490428 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:53.491152 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:53.491156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:53.491157 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:02:53.491332 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:53.491362 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:02:53.491373 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:53.491443 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:53.491517 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 03:02:53.491627 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":82,"threads_started":1}
I20260924 03:02:53.492652 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:02:53.525504 5698 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:02:53.526216 5698 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:02:53.539988 5698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8dc118972aca44f4901844e9c877147f: Generated new TSK 0
I20260924 03:02:53.540212 5698 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 03:02:53.565583 5716 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 03:02:53.565694 5716 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 03:02:53.566005 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:53.545774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53094 (local address 127.0.98.62:42193)
0924 03:02:53.545844 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:53.545846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:53.545926 (+ 80us) server_negotiation.cc:407] Connection header received
0924 03:02:53.546014 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:53.546015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:53.546025 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:53.546084 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0924 03:02:53.546245 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:53.546723 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:53.547886 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:53.548070 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:53.548122 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:53.548305 (+ 183us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 03:02:53.565820 (+ 17515us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 03:02:53.565944 (+ 124us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53094: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 03:02:53.566073 5708 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53094: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 03:02:53.566205 5715 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:02:53.545697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42193 (local address 127.0.0.1:53094)
0924 03:02:53.545861 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:53.545873 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:53.545947 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:53.546109 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:53.546110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:53.546112 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 03:02:53.546181 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:53.546183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:53.546819 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:53.546820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:53.547806 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:53.547809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:53.548142 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:53.548145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:53.548206 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:53.548212 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 03:02:53.565956 (+ 17744us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 03:02:53.566104 (+ 148us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42193: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":65,"threads_started":1}
I20260924 03:02:53.566636 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 5619
2026-09-24T03:02:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (261 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 03:02:53.604311 392 mini_oidc.cc:152] Starting JWKS server
I20260924 03:02:53.604977 392 webserver.cc:460] Webserver started at https://127.0.0.1:37427/ using document root <none> and password file <none>
I20260924 03:02:53.605018 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 03:02:53.605180 392 webserver.cc:460] Webserver started at http://127.0.0.1:36437/ using document root <none> and password file <none>
2026-09-24T03:02:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:02:53Z Disabled control of system clock
I20260924 03:02:53.612458 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37547
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37547
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37427/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 03:02:53.694667 5722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:53.694967 5722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:53.695011 5722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:53.696455 5722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:02:53.696527 5722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:53.696550 5722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:53.696571 5722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:02:53.696604 5722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 03:02:53.696628 5722 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 03:02:53.696657 5722 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37427/jwks.json
I20260924 03:02:53.698129 5722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37547
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37427/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:53.698424 5722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:53.698740 5722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:53.703678 5727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:53.703779 5728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:53.703788 5730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:53.703961 5722 server_base.cc:1034] running on GCE node
I20260924 03:02:53.704263 5722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:53.704558 5722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:53.705787 5722 hybrid_clock.cc:648] HybridClock initialized: now 1790218973705760 us; error 39 us; skew 500 ppm
I20260924 03:02:53.707432 5722 webserver.cc:460] Webserver started at http://127.0.98.62:37871/ using document root <none> and password file <none>
I20260924 03:02:53.707772 5722 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:53.707834 5722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:53.708012 5722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:53.709273 5722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "971cafc4d53c41da9310c63f4a95c069"
format_stamp: "Formatted at 2026-09-24 03:02:53 on dist-test-slave-wgsb"
server_key: "c7dbc854d8fb15ebfda7086a7793d37d"
server_key_iv: "cc7b56c3e895b221b0901176d852745a"
server_key_version: "encryptionkey@0"
I20260924 03:02:53.709667 5722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "971cafc4d53c41da9310c63f4a95c069"
format_stamp: "Formatted at 2026-09-24 03:02:53 on dist-test-slave-wgsb"
server_key: "c7dbc854d8fb15ebfda7086a7793d37d"
server_key_iv: "cc7b56c3e895b221b0901176d852745a"
server_key_version: "encryptionkey@0"
I20260924 03:02:53.711165 5722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:02:53.712059 5736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:53.712218 5722 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:53.712311 5722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "971cafc4d53c41da9310c63f4a95c069"
format_stamp: "Formatted at 2026-09-24 03:02:53 on dist-test-slave-wgsb"
server_key: "c7dbc854d8fb15ebfda7086a7793d37d"
server_key_iv: "cc7b56c3e895b221b0901176d852745a"
server_key_version: "encryptionkey@0"
I20260924 03:02:53.712368 5722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:53.731719 5722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:53.732004 5722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:53.732115 5722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:53.735693 5722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37547
I20260924 03:02:53.735762 5788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37547 every 8 connection(s)
I20260924 03:02:53.736078 5722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:02:53.736754 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 5722
I20260924 03:02:53.736860 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:02:53.736891 5789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:53.737057 392 external_mini_cluster.cc:1428] Setting key edf1e27ef2d13fc1d78d22405db9f957
I20260924 03:02:53.740134 5789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 971cafc4d53c41da9310c63f4a95c069: Bootstrap starting.
I20260924 03:02:53.740712 5789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 971cafc4d53c41da9310c63f4a95c069: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:53.741011 5789 log.cc:824] T 00000000000000000000000000000000 P 971cafc4d53c41da9310c63f4a95c069: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:53.741915 5789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 971cafc4d53c41da9310c63f4a95c069: No bootstrap required, opened a new log
I20260924 03:02:53.743291 5792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:53.737594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59974 (local address 127.0.98.62:37547)
0924 03:02:53.737821 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:53.737836 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:53.737861 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:02:53.738378 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:53.738385 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:53.738449 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:53.738713 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:53.738894 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:53.739940 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:53.740860 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:53.741507 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:53.741616 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:53.742418 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:53.742422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:53.742422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:02:53.742744 (+ 322us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:53.742784 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:02:53.742796 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:53.742864 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:53.743003 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 03:02:53.743117 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":95,"threads_started":1}
I20260924 03:02:53.744369 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:02:53.744315 5789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 971cafc4d53c41da9310c63f4a95c069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971cafc4d53c41da9310c63f4a95c069" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37547 } }
I20260924 03:02:53.744495 5789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 971cafc4d53c41da9310c63f4a95c069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:53.744542 5789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 971cafc4d53c41da9310c63f4a95c069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971cafc4d53c41da9310c63f4a95c069, State: Initialized, Role: FOLLOWER
I20260924 03:02:53.744664 5789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 971cafc4d53c41da9310c63f4a95c069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971cafc4d53c41da9310c63f4a95c069" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37547 } }
I20260924 03:02:53.744763 5789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 971cafc4d53c41da9310c63f4a95c069 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:02:53.744807 5789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 971cafc4d53c41da9310c63f4a95c069 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:02:53.744856 5789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 971cafc4d53c41da9310c63f4a95c069 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:53.745574 5789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 971cafc4d53c41da9310c63f4a95c069 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971cafc4d53c41da9310c63f4a95c069" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37547 } }
I20260924 03:02:53.745704 5789 leader_election.cc:302] T 00000000000000000000000000000000 P 971cafc4d53c41da9310c63f4a95c069 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 971cafc4d53c41da9310c63f4a95c069; no voters:
I20260924 03:02:53.745894 5789 leader_election.cc:288] T 00000000000000000000000000000000 P 971cafc4d53c41da9310c63f4a95c069 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:02:53.745952 5794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 971cafc4d53c41da9310c63f4a95c069 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:53.746127 5794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 971cafc4d53c41da9310c63f4a95c069 [term 1 LEADER]: Becoming Leader. State: Replica: 971cafc4d53c41da9310c63f4a95c069, State: Running, Role: LEADER
I20260924 03:02:53.746208 5789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 971cafc4d53c41da9310c63f4a95c069 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:02:53.746213 5794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 971cafc4d53c41da9310c63f4a95c069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971cafc4d53c41da9310c63f4a95c069" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37547 } }
I20260924 03:02:53.746659 5795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 971cafc4d53c41da9310c63f4a95c069 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "971cafc4d53c41da9310c63f4a95c069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971cafc4d53c41da9310c63f4a95c069" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37547 } } }
I20260924 03:02:53.746830 5795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 971cafc4d53c41da9310c63f4a95c069 [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:53.746683 5796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 971cafc4d53c41da9310c63f4a95c069 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 971cafc4d53c41da9310c63f4a95c069. Latest consensus state: current_term: 1 leader_uuid: "971cafc4d53c41da9310c63f4a95c069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971cafc4d53c41da9310c63f4a95c069" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37547 } } }
I20260924 03:02:53.747099 5796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 971cafc4d53c41da9310c63f4a95c069 [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:53.747123 5803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:02:53.747653 5803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:02:53.749259 5803 catalog_manager.cc:1347] Generated new cluster ID: 17a04be5c0774ed8b47771b0064ae6c1
I20260924 03:02:53.749320 5803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:02:53.757700 5803 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:02:53.758399 5803 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:02:53.773330 5803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 971cafc4d53c41da9310c63f4a95c069: Generated new TSK 0
I20260924 03:02:53.773546 5803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:02:53.786087 5792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:53.782263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59976 (local address 127.0.98.62:37547)
0924 03:02:53.782325 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:53.782328 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:53.782434 (+ 106us) server_negotiation.cc:407] Connection header received
0924 03:02:53.782506 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:53.782506 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:53.782517 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:53.782579 (+ 62us) server_negotiation.cc:226] Negotiated authn=JWT
0924 03:02:53.782739 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:53.783216 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:53.784422 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:53.784644 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:53.784728 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:53.784928 (+ 200us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 03:02:53.785884 (+ 956us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37427/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 03:02:53.786013 (+ 129us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59976: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37427/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":26}
W20260924 03:02:53.786211 5792 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59976: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37427/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 03:02:53.786211 5818 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:02:53.782173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37547 (local address 127.0.0.1:59976)
0924 03:02:53.782348 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:53.782361 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:53.782436 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:53.782602 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:53.782603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:53.782605 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 03:02:53.782668 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:53.782670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:53.783300 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:53.783301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:53.784313 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:53.784316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:53.784742 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:53.784751 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:53.784819 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:53.784825 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 03:02:53.785998 (+ 1173us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37427/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 03:02:53.786102 (+ 104us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37547: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37427/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":98,"threads_started":1}
I20260924 03:02:53.786618 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 5722
E20260924 03:02:53.797508 5819 webserver.cc:533] Webserver: sslize failed:
E20260924 03:02:53.797595 5819 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T03:02:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (220 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T03:02:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:02:53Z Disabled control of system clock
I20260924 03:02:53.831926 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38037
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38037
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218948286999-392-0/testchainca.cert with env {}
W20260924 03:02:53.914235 5823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:53.914453 5823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:53.914475 5823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:53.915653 5823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:02:53.915712 5823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:53.915727 5823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:53.915740 5823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:02:53.915752 5823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 03:02:53.915784 5823 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 03:02:53.915814 5823 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 03:02:53.917316 5823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38037
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218948286999-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:53.917589 5823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:53.917819 5823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:53.922930 5831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:53.922955 5823 server_base.cc:1034] running on GCE node
W20260924 03:02:53.922963 5829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:53.923163 5828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:53.923422 5823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:53.923789 5823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:53.924997 5823 hybrid_clock.cc:648] HybridClock initialized: now 1790218973924982 us; error 46 us; skew 500 ppm
I20260924 03:02:53.926441 5823 webserver.cc:460] Webserver started at http://127.0.98.62:41923/ using document root <none> and password file <none>
I20260924 03:02:53.926630 5823 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:53.926669 5823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:53.926770 5823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:53.927875 5823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "6c30f78a1b7c4b12b22dc04dc1abcba3"
format_stamp: "Formatted at 2026-09-24 03:02:53 on dist-test-slave-wgsb"
server_key: "ca730c989a68423f2c021159ad74b0d9"
server_key_iv: "4fcf5dd050270fc06f7501dfc6eb9f04"
server_key_version: "encryptionkey@0"
I20260924 03:02:53.928188 5823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "6c30f78a1b7c4b12b22dc04dc1abcba3"
format_stamp: "Formatted at 2026-09-24 03:02:53 on dist-test-slave-wgsb"
server_key: "ca730c989a68423f2c021159ad74b0d9"
server_key_iv: "4fcf5dd050270fc06f7501dfc6eb9f04"
server_key_version: "encryptionkey@0"
I20260924 03:02:53.929414 5823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:02:53.930131 5837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:53.930373 5823 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:53.930459 5823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "6c30f78a1b7c4b12b22dc04dc1abcba3"
format_stamp: "Formatted at 2026-09-24 03:02:53 on dist-test-slave-wgsb"
server_key: "ca730c989a68423f2c021159ad74b0d9"
server_key_iv: "4fcf5dd050270fc06f7501dfc6eb9f04"
server_key_version: "encryptionkey@0"
I20260924 03:02:53.930555 5823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:53.939324 5823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:53.939560 5823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:53.939736 5823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:53.943501 5823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38037
I20260924 03:02:53.943531 5889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38037 every 8 connection(s)
I20260924 03:02:53.943890 5823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:02:53.944885 5890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:53.946287 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 5823
I20260924 03:02:53.946388 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:02:53.946579 392 external_mini_cluster.cc:1428] Setting key e05926b2b042681506283b73875e9af3
I20260924 03:02:53.947758 5890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c30f78a1b7c4b12b22dc04dc1abcba3: Bootstrap starting.
I20260924 03:02:53.948450 5890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c30f78a1b7c4b12b22dc04dc1abcba3: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:53.948738 5890 log.cc:824] T 00000000000000000000000000000000 P 6c30f78a1b7c4b12b22dc04dc1abcba3: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:53.949453 5890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c30f78a1b7c4b12b22dc04dc1abcba3: No bootstrap required, opened a new log
I20260924 03:02:53.951788 5890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c30f78a1b7c4b12b22dc04dc1abcba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c30f78a1b7c4b12b22dc04dc1abcba3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38037 } }
I20260924 03:02:53.951920 5890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c30f78a1b7c4b12b22dc04dc1abcba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:53.951941 5890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c30f78a1b7c4b12b22dc04dc1abcba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c30f78a1b7c4b12b22dc04dc1abcba3, State: Initialized, Role: FOLLOWER
I20260924 03:02:53.952072 5890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c30f78a1b7c4b12b22dc04dc1abcba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c30f78a1b7c4b12b22dc04dc1abcba3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38037 } }
I20260924 03:02:53.952169 5890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c30f78a1b7c4b12b22dc04dc1abcba3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:02:53.952209 5890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c30f78a1b7c4b12b22dc04dc1abcba3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:02:53.952251 5890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c30f78a1b7c4b12b22dc04dc1abcba3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:53.952478 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:53.947131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53414 (local address 127.0.98.62:38037)
0924 03:02:53.947333 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:53.947340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:53.947365 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:02:53.947827 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:53.947836 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:53.947936 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:53.948054 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:53.948312 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:53.949267 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:53.950342 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:53.950747 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:53.950843 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:53.951562 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:53.951566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:53.951566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:02:53.951870 (+ 304us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:53.951903 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:02:53.951935 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:53.952056 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:53.952203 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 03:02:53.952288 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
I20260924 03:02:53.952924 5890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c30f78a1b7c4b12b22dc04dc1abcba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c30f78a1b7c4b12b22dc04dc1abcba3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38037 } }
I20260924 03:02:53.953047 5890 leader_election.cc:302] T 00000000000000000000000000000000 P 6c30f78a1b7c4b12b22dc04dc1abcba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c30f78a1b7c4b12b22dc04dc1abcba3; no voters:
I20260924 03:02:53.953191 5890 leader_election.cc:288] T 00000000000000000000000000000000 P 6c30f78a1b7c4b12b22dc04dc1abcba3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:02:53.953236 5895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c30f78a1b7c4b12b22dc04dc1abcba3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:53.953369 5895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c30f78a1b7c4b12b22dc04dc1abcba3 [term 1 LEADER]: Becoming Leader. State: Replica: 6c30f78a1b7c4b12b22dc04dc1abcba3, State: Running, Role: LEADER
I20260924 03:02:53.953404 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:02:53.953433 5890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c30f78a1b7c4b12b22dc04dc1abcba3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:02:53.953454 5895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c30f78a1b7c4b12b22dc04dc1abcba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c30f78a1b7c4b12b22dc04dc1abcba3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38037 } }
I20260924 03:02:53.953790 5895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c30f78a1b7c4b12b22dc04dc1abcba3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c30f78a1b7c4b12b22dc04dc1abcba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c30f78a1b7c4b12b22dc04dc1abcba3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38037 } } }
I20260924 03:02:53.953867 5895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c30f78a1b7c4b12b22dc04dc1abcba3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:53.953987 5895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c30f78a1b7c4b12b22dc04dc1abcba3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c30f78a1b7c4b12b22dc04dc1abcba3. Latest consensus state: current_term: 1 leader_uuid: "6c30f78a1b7c4b12b22dc04dc1abcba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c30f78a1b7c4b12b22dc04dc1abcba3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38037 } } }
I20260924 03:02:53.954048 5895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c30f78a1b7c4b12b22dc04dc1abcba3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:53.954172 5902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:02:53.954839 5902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:02:53.956310 5902 catalog_manager.cc:1347] Generated new cluster ID: 588d4bc3adde4dfa9548cf68c25f70a5
I20260924 03:02:53.956360 5902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:02:53.963733 5902 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:02:53.964368 5902 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:02:53.970450 5902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c30f78a1b7c4b12b22dc04dc1abcba3: Generated new TSK 0
I20260924 03:02:53.970647 5902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:02:54.046430 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:53.979819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53426 (local address 127.0.98.62:38037)
0924 03:02:53.979868 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:53.979870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:53.980028 (+ 158us) server_negotiation.cc:407] Connection header received
0924 03:02:53.980061 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:53.980062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:53.980071 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:53.980126 (+ 55us) server_negotiation.cc:226] Negotiated authn=JWT
0924 03:02:53.980286 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:53.980855 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:53.982078 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:53.982352 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:53.982407 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:53.982608 (+ 201us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 03:02:54.046220 (+ 63612us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 03:02:54.046371 (+ 151us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53426: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":27}
W20260924 03:02:54.046546 5893 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53426: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 03:02:54.046686 5919 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:02:53.979729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38037 (local address 127.0.0.1:53426)
0924 03:02:53.979913 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:53.979924 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:53.979991 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:53.980144 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:53.980145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:53.980147 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 03:02:53.980218 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:53.980220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:53.980937 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:53.980938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:53.981878 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:53.981880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:53.982432 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:53.982438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:53.982501 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:53.982506 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 03:02:54.046392 (+ 63886us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 03:02:54.046559 (+ 167us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38037: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":95,"threads_started":1}
I20260924 03:02:54.047086 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 5823
2026-09-24T03:02:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (240 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (86 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (89 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:02:54 dist-test-slave-wgsb krb5kdc[5931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:02:54 dist-test-slave-wgsb krb5kdc[5931](info): set up 2 sockets
Sep 24 03:02:54 dist-test-slave-wgsb krb5kdc[5931](info): commencing operation
krb5kdc: starting...
W20260924 03:02:55.255919 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:02:55 dist-test-slave-wgsb krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:02:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:02:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:02:55.378362 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41919
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:55.458365 5944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:55.458583 5944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:55.458606 5944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:55.459769 5944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:02:55.459810 5944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:55.459823 5944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:55.459836 5944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:02:55.459848 5944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:02:55.461387 5944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:55.461635 5944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:55.461858 5944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:55.466758 5952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:55.466789 5950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:55.466769 5949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:55.467038 5944 server_base.cc:1034] running on GCE node
I20260924 03:02:55.467309 5944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:55.467648 5944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:55.468806 5944 hybrid_clock.cc:648] HybridClock initialized: now 1790218975468805 us; error 59 us; skew 500 ppm
Sep 24 03:02:55 dist-test-slave-wgsb krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:55.477362 5944 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:02:55.477829 5944 webserver.cc:460] Webserver started at http://127.0.98.62:39727/ using document root <none> and password file <none>
I20260924 03:02:55.478071 5944 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:55.478138 5944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:55.478293 5944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:55.479429 5944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "bc0f87b254734a17babef26dd35f5752"
format_stamp: "Formatted at 2026-09-24 03:02:55 on dist-test-slave-wgsb"
server_key: "6f6f4e41b4683b5c0f55acd2950e517e"
server_key_iv: "9c61969606d540594345ed4e20d3e39e"
server_key_version: "encryptionkey@0"
I20260924 03:02:55.479832 5944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "bc0f87b254734a17babef26dd35f5752"
format_stamp: "Formatted at 2026-09-24 03:02:55 on dist-test-slave-wgsb"
server_key: "6f6f4e41b4683b5c0f55acd2950e517e"
server_key_iv: "9c61969606d540594345ed4e20d3e39e"
server_key_version: "encryptionkey@0"
I20260924 03:02:55.481202 5944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:02:55.482048 5959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:55.482254 5944 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:02:55.482384 5944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "bc0f87b254734a17babef26dd35f5752"
format_stamp: "Formatted at 2026-09-24 03:02:55 on dist-test-slave-wgsb"
server_key: "6f6f4e41b4683b5c0f55acd2950e517e"
server_key_iv: "9c61969606d540594345ed4e20d3e39e"
server_key_version: "encryptionkey@0"
I20260924 03:02:55.482461 5944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:55.494737 5944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:55.496860 5944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:55.497032 5944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:55.501307 5944 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41919
I20260924 03:02:55.501375 6011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41919 every 8 connection(s)
I20260924 03:02:55.501892 5944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:02:55.502568 6012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:55.503059 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 5944
I20260924 03:02:55.503156 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:02:55.503350 392 external_mini_cluster.cc:1428] Setting key 4545646b9e421176257f86f8bf247b54
I20260924 03:02:55.505860 6012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc0f87b254734a17babef26dd35f5752: Bootstrap starting.
I20260924 03:02:55.506462 6012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc0f87b254734a17babef26dd35f5752: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:55.506819 6012 log.cc:824] T 00000000000000000000000000000000 P bc0f87b254734a17babef26dd35f5752: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:55.507547 6012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc0f87b254734a17babef26dd35f5752: No bootstrap required, opened a new log
I20260924 03:02:55.510033 6012 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc0f87b254734a17babef26dd35f5752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0f87b254734a17babef26dd35f5752" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41919 } }
I20260924 03:02:55.510185 6012 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc0f87b254734a17babef26dd35f5752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:55.510232 6012 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc0f87b254734a17babef26dd35f5752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc0f87b254734a17babef26dd35f5752, State: Initialized, Role: FOLLOWER
Sep 24 03:02:55 dist-test-slave-wgsb krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:55.510362 6012 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc0f87b254734a17babef26dd35f5752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0f87b254734a17babef26dd35f5752" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41919 } }
I20260924 03:02:55.510435 6012 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc0f87b254734a17babef26dd35f5752 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:02:55.510493 6012 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc0f87b254734a17babef26dd35f5752 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:02:55.510545 6012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc0f87b254734a17babef26dd35f5752 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:55.511191 6012 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc0f87b254734a17babef26dd35f5752 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0f87b254734a17babef26dd35f5752" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41919 } }
I20260924 03:02:55.511320 6012 leader_election.cc:302] T 00000000000000000000000000000000 P bc0f87b254734a17babef26dd35f5752 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc0f87b254734a17babef26dd35f5752; no voters:
I20260924 03:02:55.511472 6012 leader_election.cc:288] T 00000000000000000000000000000000 P bc0f87b254734a17babef26dd35f5752 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:02:55.511507 6017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc0f87b254734a17babef26dd35f5752 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:55.511631 6017 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc0f87b254734a17babef26dd35f5752 [term 1 LEADER]: Becoming Leader. State: Replica: bc0f87b254734a17babef26dd35f5752, State: Running, Role: LEADER
I20260924 03:02:55.511756 6012 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc0f87b254734a17babef26dd35f5752 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:02:55.511757 6017 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc0f87b254734a17babef26dd35f5752 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0f87b254734a17babef26dd35f5752" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41919 } }
I20260924 03:02:55.513101 6018 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc0f87b254734a17babef26dd35f5752 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc0f87b254734a17babef26dd35f5752. Latest consensus state: current_term: 1 leader_uuid: "bc0f87b254734a17babef26dd35f5752" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0f87b254734a17babef26dd35f5752" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41919 } } }
I20260924 03:02:55.513445 6018 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc0f87b254734a17babef26dd35f5752 [sys.catalog]: This master's current role is: LEADER
W20260924 03:02:55.513608 6031 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc0f87b254734a17babef26dd35f5752: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:02:55.513669 6031 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:02:55.513238 6019 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc0f87b254734a17babef26dd35f5752 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc0f87b254734a17babef26dd35f5752" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0f87b254734a17babef26dd35f5752" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41919 } } }
I20260924 03:02:55.513754 6019 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc0f87b254734a17babef26dd35f5752 [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:55.513901 6034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:02:55.514083 6034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:02:55.515081 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:55.503946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41264 (local address 127.0.98.62:41919)
0924 03:02:55.504140 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:55.504149 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:55.504183 (+ 34us) server_negotiation.cc:407] Connection header received
0924 03:02:55.504583 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:55.504589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:55.504634 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:55.504751 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:55.505575 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:55.506524 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:55.507564 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:55.508063 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:55.508147 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:55.510881 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:55.510885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:55.510886 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:55.510897 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:55.513100 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:55.513781 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:55.513784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:55.513784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:55.513899 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:55.514133 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:55.514134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:55.514135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:55.514380 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:55.514535 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:55.514713 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 03:02:55.514867 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":96,"threads_started":1}
I20260924 03:02:55.515650 6034 catalog_manager.cc:1347] Generated new cluster ID: 6c4b1c5e1f2c45a89a03514dc83715f1
I20260924 03:02:55.515709 6034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:02:55.525147 6034 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:02:55.525695 6034 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:02:55.528138 6034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc0f87b254734a17babef26dd35f5752: Generated new TSK 0
I20260924 03:02:55.528282 6034 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:02:55.559237 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41919
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:55.650849 6038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:55.651085 6038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:55.651125 6038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:55.652371 6038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:55.652438 6038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:55.652483 6038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:02:55.654222 6038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:55.654506 6038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:55.654753 6038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:55.660827 6046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:55.661051 6038 server_base.cc:1034] running on GCE node
W20260924 03:02:55.660827 6043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:55.660823 6044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:55.661329 6038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:55.661576 6038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:55.662746 6038 hybrid_clock.cc:648] HybridClock initialized: now 1790218975662731 us; error 31 us; skew 500 ppm
Sep 24 03:02:55 dist-test-slave-wgsb krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:55.665426 6038 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:02:55.665823 6038 webserver.cc:460] Webserver started at http://127.0.98.1:35175/ using document root <none> and password file <none>
I20260924 03:02:55.666021 6038 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:55.666061 6038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:55.666152 6038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:55.667215 6038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-0/data/instance:
uuid: "23938c1b9cd44d56b8612f6b6fe4a037"
format_stamp: "Formatted at 2026-09-24 03:02:55 on dist-test-slave-wgsb"
server_key: "0133e50a84c4e09b5eae82dfa125f002"
server_key_iv: "b68e58219cef73feeca33ee12510ee2f"
server_key_version: "encryptionkey@0"
I20260924 03:02:55.667518 6038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-0/wal/instance:
uuid: "23938c1b9cd44d56b8612f6b6fe4a037"
format_stamp: "Formatted at 2026-09-24 03:02:55 on dist-test-slave-wgsb"
server_key: "0133e50a84c4e09b5eae82dfa125f002"
server_key_iv: "b68e58219cef73feeca33ee12510ee2f"
server_key_version: "encryptionkey@0"
I20260924 03:02:55.668850 6038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:02:55.669590 6053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:55.669834 6038 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:02:55.669921 6038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "23938c1b9cd44d56b8612f6b6fe4a037"
format_stamp: "Formatted at 2026-09-24 03:02:55 on dist-test-slave-wgsb"
server_key: "0133e50a84c4e09b5eae82dfa125f002"
server_key_iv: "b68e58219cef73feeca33ee12510ee2f"
server_key_version: "encryptionkey@0"
I20260924 03:02:55.669978 6038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:55.686201 6038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:55.688503 6038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:55.688616 6038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:55.688800 6038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:55.689143 6038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:55.689177 6038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:55.689240 6038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:55.689270 6038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:55.695021 6038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44039
I20260924 03:02:55.695097 6166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44039 every 8 connection(s)
I20260924 03:02:55.695426 6038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 03:02:55 dist-test-slave-wgsb krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:55.704660 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:55.696177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37227 (local address 127.0.98.62:41919)
0924 03:02:55.696221 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:55.696223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:55.696450 (+ 227us) server_negotiation.cc:407] Connection header received
0924 03:02:55.697349 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:55.697350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:55.697363 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:55.697417 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:55.697975 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:55.698471 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:55.699478 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:55.699856 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:55.699919 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:55.702106 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:55.702107 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:55.702108 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:55.702114 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:55.703410 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:55.703870 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:55.703871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:55.703871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:55.703921 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:55.704131 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:55.704131 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:55.704131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:55.704232 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:55.704277 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:55.704527 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 03:02:55.704593 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:02:55.704974 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 6038
I20260924 03:02:55.704988 6169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:55.696166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41919 (local address 127.0.98.1:37227)
0924 03:02:55.696316 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:55.696350 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:55.697248 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:55.697469 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:55.697472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:55.697712 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:55.697900 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:55.697903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:55.698562 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:55.698563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:55.699386 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:55.699388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:55.699953 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:55.699954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:55.700056 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:55.700663 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:55.700671 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:55.702023 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:55.703515 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:55.703521 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:55.703522 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:55.703782 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:55.703992 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:55.703993 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:55.703993 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:55.704088 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:55.704291 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:55.704295 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:55.704406 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 03:02:55.704491 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 03:02:55.704604 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":93,"threads_started":1}
I20260924 03:02:55.705090 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:02:55.705261 392 external_mini_cluster.cc:1428] Setting key 2b19cf20aeeecab17484a8f58b0fda28
I20260924 03:02:55.705752 6167 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41919
I20260924 03:02:55.705849 6167 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:55.706045 6167 heartbeater.cc:511] Master 127.0.98.62:41919 requested a full tablet report, sending...
I20260924 03:02:55.706588 5975 ts_manager.cc:194] Registered new tserver with Master: 23938c1b9cd44d56b8612f6b6fe4a037 (127.0.98.1:44039)
I20260924 03:02:55.707233 5975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37227
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:02:55.731681 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41919
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:55.814929 6172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:55.815155 6172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:55.815181 6172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:55.816407 6172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:55.816455 6172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:55.816484 6172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:02:55.817972 6172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:55.818228 6172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:55.818468 6172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:55.823571 6177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:55.823568 6178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:55.823557 6180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:55.823995 6172 server_base.cc:1034] running on GCE node
I20260924 03:02:55.824235 6172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:55.824484 6172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:55.825690 6172 hybrid_clock.cc:648] HybridClock initialized: now 1790218975825679 us; error 38 us; skew 500 ppm
Sep 24 03:02:55 dist-test-slave-wgsb krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:55.828521 6172 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:02:55.829025 6172 webserver.cc:460] Webserver started at http://127.0.98.2:43365/ using document root <none> and password file <none>
I20260924 03:02:55.829294 6172 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:55.829345 6172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:55.829485 6172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:55.830606 6172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-1/data/instance:
uuid: "cbd7112966464126be5a85dab6771f88"
format_stamp: "Formatted at 2026-09-24 03:02:55 on dist-test-slave-wgsb"
server_key: "0f87a98d6a6a7d63056cf3e6203e618f"
server_key_iv: "544252afde423f94f05bd4f917196151"
server_key_version: "encryptionkey@0"
I20260924 03:02:55.831002 6172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cbd7112966464126be5a85dab6771f88"
format_stamp: "Formatted at 2026-09-24 03:02:55 on dist-test-slave-wgsb"
server_key: "0f87a98d6a6a7d63056cf3e6203e618f"
server_key_iv: "544252afde423f94f05bd4f917196151"
server_key_version: "encryptionkey@0"
I20260924 03:02:55.832506 6172 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:02:55.833343 6187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:55.833607 6172 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:55.833774 6172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "cbd7112966464126be5a85dab6771f88"
format_stamp: "Formatted at 2026-09-24 03:02:55 on dist-test-slave-wgsb"
server_key: "0f87a98d6a6a7d63056cf3e6203e618f"
server_key_iv: "544252afde423f94f05bd4f917196151"
server_key_version: "encryptionkey@0"
I20260924 03:02:55.833864 6172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:55.852151 6172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:55.854267 6172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:55.854442 6172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:55.854720 6172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:55.855039 6172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:55.855098 6172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:55.855139 6172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:55.855170 6172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:55.861279 6172 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42203
I20260924 03:02:55.861320 6300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42203 every 8 connection(s)
I20260924 03:02:55.861629 6172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:02:55.866600 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 6172
I20260924 03:02:55.866695 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:02:55.866910 392 external_mini_cluster.cc:1428] Setting key 25ad83a7404057492f46d9cc0a144ba5
Sep 24 03:02:55 dist-test-slave-wgsb krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:55.871322 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:55.862356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49657 (local address 127.0.98.62:41919)
0924 03:02:55.862417 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:55.862419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:55.862641 (+ 222us) server_negotiation.cc:407] Connection header received
0924 03:02:55.863639 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:55.863640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:55.863653 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:55.863726 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:55.864320 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:55.864842 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:55.865876 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:55.866054 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:55.866110 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:55.868469 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:55.868470 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:55.868470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:55.868478 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:55.869895 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:55.870330 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:55.870331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:55.870332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:55.870396 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:55.870658 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:55.870659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:55.870659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:55.870782 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:55.870853 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:55.871140 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 03:02:55.871230 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 03:02:55.871527 6303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:55.862338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41919 (local address 127.0.98.2:49657)
0924 03:02:55.862508 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:55.862550 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:55.863502 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:55.863784 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:55.863787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:55.864036 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:55.864234 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:55.864238 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:55.864930 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:55.864930 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:55.865799 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:55.865801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:55.866149 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:55.866150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:55.866270 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:55.866833 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:55.866844 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:55.868369 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:55.869998 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:55.870006 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:55.870008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:55.870239 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:55.870461 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:55.870462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:55.870463 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:55.870582 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:55.870856 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:55.870862 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:55.871006 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 03:02:55.871118 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 03:02:55.871281 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":96,"threads_started":1}
I20260924 03:02:55.872025 6301 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41919
I20260924 03:02:55.872107 6301 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:55.872320 6301 heartbeater.cc:511] Master 127.0.98.62:41919 requested a full tablet report, sending...
I20260924 03:02:55.872741 5975 ts_manager.cc:194] Registered new tserver with Master: cbd7112966464126be5a85dab6771f88 (127.0.98.2:42203)
I20260924 03:02:55.873165 5975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49657
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:02:55.894637 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41919
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:55.976207 6306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:55.976409 6306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:55.976433 6306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:55.977594 6306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:55.977641 6306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:55.977667 6306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:02:55.979265 6306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:55.979506 6306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:55.979780 6306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:55.984773 6314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:55.984773 6312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:55.984833 6311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:55.985062 6306 server_base.cc:1034] running on GCE node
I20260924 03:02:55.985337 6306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:55.985581 6306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:55.986727 6306 hybrid_clock.cc:648] HybridClock initialized: now 1790218975986716 us; error 45 us; skew 500 ppm
Sep 24 03:02:55 dist-test-slave-wgsb krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:55.989326 6306 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:02:55.989784 6306 webserver.cc:460] Webserver started at http://127.0.98.3:46805/ using document root <none> and password file <none>
I20260924 03:02:55.990018 6306 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:55.990062 6306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:55.990228 6306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:55.991261 6306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-2/data/instance:
uuid: "99e072deedd6440c85ade04bda077cff"
format_stamp: "Formatted at 2026-09-24 03:02:55 on dist-test-slave-wgsb"
server_key: "f25ef67a6a0a987a346a34ab2b88be16"
server_key_iv: "11e8c9a8c430467aadd71547d2d3fd2e"
server_key_version: "encryptionkey@0"
I20260924 03:02:55.991662 6306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-2/wal/instance:
uuid: "99e072deedd6440c85ade04bda077cff"
format_stamp: "Formatted at 2026-09-24 03:02:55 on dist-test-slave-wgsb"
server_key: "f25ef67a6a0a987a346a34ab2b88be16"
server_key_iv: "11e8c9a8c430467aadd71547d2d3fd2e"
server_key_version: "encryptionkey@0"
I20260924 03:02:55.992971 6306 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:02:55.993711 6321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:55.993891 6306 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:02:55.993953 6306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "99e072deedd6440c85ade04bda077cff"
format_stamp: "Formatted at 2026-09-24 03:02:55 on dist-test-slave-wgsb"
server_key: "f25ef67a6a0a987a346a34ab2b88be16"
server_key_iv: "11e8c9a8c430467aadd71547d2d3fd2e"
server_key_version: "encryptionkey@0"
I20260924 03:02:55.994035 6306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:56.007586 6306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:56.009584 6306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:56.009686 6306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:56.009943 6306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:56.010244 6306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:56.010277 6306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:56.010298 6306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:56.010345 6306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:56.017282 6306 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41131
I20260924 03:02:56.017318 6434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41131 every 8 connection(s)
I20260924 03:02:56.017673 6306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:02:56.019651 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 6306
I20260924 03:02:56.019745 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:02:56.019963 392 external_mini_cluster.cc:1428] Setting key d874dc504020b2501e401e8101a2943c
Sep 24 03:02:56 dist-test-slave-wgsb krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:02:56.027312 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:56.018447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42685 (local address 127.0.98.62:41919)
0924 03:02:56.018527 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:56.018529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:56.018685 (+ 156us) server_negotiation.cc:407] Connection header received
0924 03:02:56.019674 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:56.019676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:56.019691 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:56.019766 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:56.020600 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:56.021212 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:56.022237 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:56.022456 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:56.022532 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:56.024688 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:56.024689 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:56.024689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:56.024697 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:56.025975 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:56.026368 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:56.026369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:56.026370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:56.026430 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:56.026677 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:56.026678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:56.026678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:56.026799 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:56.026876 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:56.027137 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 03:02:56.027215 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 03:02:56.027521 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:56.018397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41919 (local address 127.0.98.3:42685)
0924 03:02:56.018555 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:56.018592 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:56.019531 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:56.019845 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:56.019850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:56.020241 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:56.020525 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:56.020532 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:56.021312 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:56.021313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:56.022139 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:56.022141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:56.022536 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:56.022537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:56.022674 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:56.023157 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:56.023166 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:56.024593 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:56.026061 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:56.026068 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:56.026071 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:56.026293 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:56.026496 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:56.026497 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:56.026498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:56.026609 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:56.026866 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:56.026872 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:56.027011 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 03:02:56.027119 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 03:02:56.027289 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":98,"threads_started":1}
I20260924 03:02:56.028053 6435 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41919
I20260924 03:02:56.028183 6435 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:56.028383 6435 heartbeater.cc:511] Master 127.0.98.62:41919 requested a full tablet report, sending...
I20260924 03:02:56.028753 5975 ts_manager.cc:194] Registered new tserver with Master: 99e072deedd6440c85ade04bda077cff (127.0.98.3:41131)
I20260924 03:02:56.029101 5975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42685
I20260924 03:02:56.031687 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/test-admin.keytab.
Sep 24 03:02:56 dist-test-slave-wgsb krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:56.049391 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 03:02:56.051187 6442 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:02:56.050336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37710 (local address 127.0.98.1:44039)
0924 03:02:56.050518 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:56.050523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:56.050532 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:02:56.050567 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:56.050569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:56.050582 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:56.050689 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:56.051072 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37710: BlockingRecv error: recv got EOF from 127.0.0.1:37710 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":87,"threads_started":1}
W20260924 03:02:56.051304 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 03:02:56 dist-test-slave-wgsb krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:56.052335 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 03:02:56 dist-test-slave-wgsb krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:56.053224 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 03:02:56 dist-test-slave-wgsb krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 03:02:56.061025 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:56.053745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37726 (local address 127.0.98.1:44039)
0924 03:02:56.053783 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:56.053786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:56.053865 (+ 79us) server_negotiation.cc:407] Connection header received
0924 03:02:56.053935 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:56.053936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:56.053946 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:56.054021 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:56.054405 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:56.055151 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:56.055980 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:56.056300 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:56.056383 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:56.058361 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:56.058370 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:56.058371 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:56.058379 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:56.059876 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:56.060232 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:56.060237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:56.060237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:56.060295 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:56.060489 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:56.060490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:56.060491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:56.060620 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:56.060774 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:56.060825 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 03:02:56.060938 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 03:02:56.062637 6442 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:02:56.062156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37742 (local address 127.0.98.1:44039)
0924 03:02:56.062188 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:56.062190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:56.062242 (+ 52us) server_negotiation.cc:407] Connection header received
0924 03:02:56.062285 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:56.062286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:56.062294 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:56.062338 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:56.062571 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37742: BlockingRecv error: recv got EOF from 127.0.0.1:37742 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 03:02:56.062752 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 03:02:56 dist-test-slave-wgsb krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:56.063745 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 03:02:56 dist-test-slave-wgsb krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:56.064608 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 03:02:56 dist-test-slave-wgsb krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 03:02:56.071557 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:56.065123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37752 (local address 127.0.98.1:44039)
0924 03:02:56.065176 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:56.065178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:56.065208 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:02:56.065281 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:56.065281 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:56.065290 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:56.065419 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:56.065782 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:56.066256 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:56.066926 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:56.067131 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:56.067212 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:56.069163 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:56.069165 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:56.069165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:56.069172 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:56.070375 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:56.070740 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:56.070741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:56.070741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:56.070799 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:56.071065 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:56.071066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:56.071067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:56.071167 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:56.071263 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:56.071383 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 03:02:56.071462 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260924 03:02:56.073213 6442 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:02:56.072690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37768 (local address 127.0.98.1:44039)
0924 03:02:56.072723 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:56.072724 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:56.072785 (+ 61us) server_negotiation.cc:407] Connection header received
0924 03:02:56.072827 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:56.072827 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:56.072834 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:56.072885 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:56.073162 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37768: BlockingRecv error: recv got EOF from 127.0.0.1:37768 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 03:02:56.073413 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218948286999-392-0/krb5kdc/krb5cc)
Sep 24 03:02:56 dist-test-slave-wgsb krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:56.074415 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 03:02:56 dist-test-slave-wgsb krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:56.075225 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 03:02:56 dist-test-slave-wgsb krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 03:02:56.082451 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:56.075687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37774 (local address 127.0.98.1:44039)
0924 03:02:56.075719 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:56.075720 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:56.075841 (+ 121us) server_negotiation.cc:407] Connection header received
0924 03:02:56.075914 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:56.075915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:56.075924 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:56.075979 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:56.076362 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:56.076821 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:56.077513 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:56.077702 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:56.077774 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:56.080045 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:56.080046 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:56.080047 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:56.080053 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:56.081411 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:56.081800 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:56.081801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:56.081801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:56.081854 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:56.082081 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:56.082081 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:56.082082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:56.082167 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:56.082239 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:56.082340 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 03:02:56.082391 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:02:56.083120 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 6038
I20260924 03:02:56.088848 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 6172
I20260924 03:02:56.093310 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 6306
I20260924 03:02:56.097929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 5944
2026-09-24T03:02:56Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1877 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:02:56 dist-test-slave-wgsb krb5kdc[6455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:02:56 dist-test-slave-wgsb krb5kdc[6455](info): set up 2 sockets
Sep 24 03:02:56 dist-test-slave-wgsb krb5kdc[6455](info): commencing operation
krb5kdc: starting...
W20260924 03:02:57.171989 392 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:02:57 dist-test-slave-wgsb krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:02:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:02:57Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 03:02:57.299463 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36379
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36379
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:57.379978 6468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:57.380187 6468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:57.380218 6468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:57.381323 6468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:02:57.381361 6468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:57.381376 6468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:57.381388 6468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:02:57.381400 6468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:02:57.382825 6468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:57.383026 6468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:57.383198 6468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:57.388245 6473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:57.388273 6474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:57.388540 6476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:57.388672 6468 server_base.cc:1034] running on GCE node
I20260924 03:02:57.388945 6468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:57.389243 6468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:57.390439 6468 hybrid_clock.cc:648] HybridClock initialized: now 1790218977390422 us; error 44 us; skew 500 ppm
Sep 24 03:02:57 dist-test-slave-wgsb krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218977, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:57.393131 6468 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 03:02:57.393597 6468 webserver.cc:460] Webserver started at http://127.0.98.62:34783/ using document root <none> and password file <none>
I20260924 03:02:57.393833 6468 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:57.393903 6468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:57.394057 6468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:57.395150 6468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "105f4753f35f455c9ebfdf86ba7c065c"
format_stamp: "Formatted at 2026-09-24 03:02:57 on dist-test-slave-wgsb"
server_key: "ab41a4f4032229640ab6a9a79c3f97a3"
server_key_iv: "99077fba6340d3ee3b9afb7de7d9ec15"
server_key_version: "encryptionkey@0"
I20260924 03:02:57.395531 6468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "105f4753f35f455c9ebfdf86ba7c065c"
format_stamp: "Formatted at 2026-09-24 03:02:57 on dist-test-slave-wgsb"
server_key: "ab41a4f4032229640ab6a9a79c3f97a3"
server_key_iv: "99077fba6340d3ee3b9afb7de7d9ec15"
server_key_version: "encryptionkey@0"
I20260924 03:02:57.396950 6468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:02:57.397848 6483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:57.398043 6468 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:02:57.398167 6468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "105f4753f35f455c9ebfdf86ba7c065c"
format_stamp: "Formatted at 2026-09-24 03:02:57 on dist-test-slave-wgsb"
server_key: "ab41a4f4032229640ab6a9a79c3f97a3"
server_key_iv: "99077fba6340d3ee3b9afb7de7d9ec15"
server_key_version: "encryptionkey@0"
I20260924 03:02:57.398252 6468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:57.411219 6468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:57.413254 6468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:57.413435 6468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:57.417582 6468 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36379
I20260924 03:02:57.417614 6535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36379 every 8 connection(s)
I20260924 03:02:57.417982 6468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:02:57.419064 6536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:57.421452 6536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c: Bootstrap starting.
I20260924 03:02:57.422014 6536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:57.422258 6536 log.cc:824] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:57.422997 6536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c: No bootstrap required, opened a new log
I20260924 03:02:57.424243 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 6468
I20260924 03:02:57.424363 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:02:57.424564 392 external_mini_cluster.cc:1428] Setting key 816b8ede2908034e209c838db615bd89
I20260924 03:02:57.424863 6536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } }
I20260924 03:02:57.425081 6536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:57.425137 6536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 105f4753f35f455c9ebfdf86ba7c065c, State: Initialized, Role: FOLLOWER
I20260924 03:02:57.425310 6536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } }
I20260924 03:02:57.425405 6536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:02:57.425442 6536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:02:57.425513 6536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:57.426707 6536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } }
I20260924 03:02:57.426795 6536 leader_election.cc:302] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 105f4753f35f455c9ebfdf86ba7c065c; no voters:
I20260924 03:02:57.426973 6536 leader_election.cc:288] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:02:57.427018 6541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:57.427160 6541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [term 1 LEADER]: Becoming Leader. State: Replica: 105f4753f35f455c9ebfdf86ba7c065c, State: Running, Role: LEADER
I20260924 03:02:57.427255 6536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:02:57.427295 6541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } }
I20260924 03:02:57.427635 6541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 105f4753f35f455c9ebfdf86ba7c065c. Latest consensus state: current_term: 1 leader_uuid: "105f4753f35f455c9ebfdf86ba7c065c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } }
I20260924 03:02:57.427641 6543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "105f4753f35f455c9ebfdf86ba7c065c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } }
I20260924 03:02:57.427829 6541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:57.427874 6543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:57.428155 6550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:02:57.428773 6550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:02:57.431255 6550 catalog_manager.cc:1347] Generated new cluster ID: 66173977c15d45668f6c482930c04bf9
I20260924 03:02:57.431326 6550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:02:57 dist-test-slave-wgsb krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:02:57.436113 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:57.425160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54162 (local address 127.0.98.62:36379)
0924 03:02:57.425365 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:57.425373 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:57.425431 (+ 58us) server_negotiation.cc:407] Connection header received
0924 03:02:57.425564 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:57.425572 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:57.425632 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:57.425745 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:57.426590 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:57.427880 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:57.429021 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:57.429592 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:57.429728 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:57.432462 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:57.432466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:57.432467 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:57.432478 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:57.434409 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:57.434826 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:57.434829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:57.434830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:57.434960 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:57.435177 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:57.435177 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:57.435178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:57.435466 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:57.435633 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:57.435777 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 03:02:57.435918 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":104,"threads_started":1}
I20260924 03:02:57.439719 6550 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:02:57.440448 6550 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:02:57.447503 6550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c: Generated new TSK 0
I20260924 03:02:57.447724 6550 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 03:02:57.476018 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36379
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:57.561156 6562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:57.561398 6562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:57.561436 6562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:57.562608 6562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:57.562683 6562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:57.562726 6562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:02:57.564411 6562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:57.564713 6562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:57.564965 6562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:57.570518 6568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:57.570520 6570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:57.570544 6562 server_base.cc:1034] running on GCE node
W20260924 03:02:57.570658 6567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:57.570953 6562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:57.571236 6562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:57.572412 6562 hybrid_clock.cc:648] HybridClock initialized: now 1790218977572365 us; error 45 us; skew 500 ppm
Sep 24 03:02:57 dist-test-slave-wgsb krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218977, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:57.575373 6562 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 03:02:57.575922 6562 webserver.cc:460] Webserver started at http://127.0.98.1:35349/ using document root <none> and password file <none>
I20260924 03:02:57.576164 6562 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:57.576210 6562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:57.576349 6562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:57.577486 6562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/data/instance:
uuid: "7bce5a68994249e7884543c4c7276d3f"
format_stamp: "Formatted at 2026-09-24 03:02:57 on dist-test-slave-wgsb"
server_key: "cb7bcf987ba551ee2b15ff670afc2213"
server_key_iv: "5b5cc3982bf652b153474dc824457491"
server_key_version: "encryptionkey@0"
I20260924 03:02:57.577850 6562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7bce5a68994249e7884543c4c7276d3f"
format_stamp: "Formatted at 2026-09-24 03:02:57 on dist-test-slave-wgsb"
server_key: "cb7bcf987ba551ee2b15ff670afc2213"
server_key_iv: "5b5cc3982bf652b153474dc824457491"
server_key_version: "encryptionkey@0"
I20260924 03:02:57.579290 6562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:02:57.580192 6577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:57.580386 6562 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 03:02:57.580521 6562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "7bce5a68994249e7884543c4c7276d3f"
format_stamp: "Formatted at 2026-09-24 03:02:57 on dist-test-slave-wgsb"
server_key: "cb7bcf987ba551ee2b15ff670afc2213"
server_key_iv: "5b5cc3982bf652b153474dc824457491"
server_key_version: "encryptionkey@0"
I20260924 03:02:57.580596 6562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:57.597471 6562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:57.599977 6562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:57.600092 6562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:57.600281 6562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:57.600605 6562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:57.600641 6562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:57.600663 6562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:57.600711 6562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:57.606961 6562 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33289
I20260924 03:02:57.607003 6690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33289 every 8 connection(s)
I20260924 03:02:57.607347 6562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:02:57.611105 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 6562
I20260924 03:02:57.611250 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:02:57.611450 392 external_mini_cluster.cc:1428] Setting key e151e5b2518f7bc4013fd54d20d60839
Sep 24 03:02:57 dist-test-slave-wgsb krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218977, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:02:57.617728 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:57.608122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56535 (local address 127.0.98.62:36379)
0924 03:02:57.608171 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:57.608173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:57.608362 (+ 189us) server_negotiation.cc:407] Connection header received
0924 03:02:57.608941 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:57.608942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:57.608957 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:57.609012 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:57.610026 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:57.610540 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:57.611709 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:57.611938 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:57.612011 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:57.614273 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:57.614275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:57.614276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:57.614285 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:57.616034 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:57.616573 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:57.616574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:57.616574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:57.616631 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:57.616962 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:57.616963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:57.616963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:57.617105 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:57.617174 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:57.617564 (+ 390us) server_negotiation.cc:299] Negotiation successful
0924 03:02:57.617651 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:02:57.617991 6693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:57.608078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:56535)
0924 03:02:57.608224 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:57.608272 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:57.608792 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:57.609082 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:57.609086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:57.609715 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:57.609916 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:57.609920 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:57.610636 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:57.610637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:57.611577 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:57.611579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:57.612093 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:57.612094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:57.612240 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:57.612799 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:57.612808 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:57.614174 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:57.616129 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:57.616138 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:57.616141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:57.616483 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:57.616703 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:57.616704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:57.616704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:57.616865 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:57.617187 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:57.617195 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:57.617391 (+ 196us) client_negotiation.cc:769] Sending connection context
0924 03:02:57.617527 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 03:02:57.617723 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":73,"threads_started":1}
I20260924 03:02:57.618574 6691 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36379
I20260924 03:02:57.618702 6691 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:57.618891 6691 heartbeater.cc:511] Master 127.0.98.62:36379 requested a full tablet report, sending...
I20260924 03:02:57.619473 6500 ts_manager.cc:194] Registered new tserver with Master: 7bce5a68994249e7884543c4c7276d3f (127.0.98.1:33289)
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 03:02:57.620160 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 03:02:57.640547 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36379
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:57.721109 6696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:57.721313 6696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:57.721338 6696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:57.722515 6696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:57.722561 6696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:57.722589 6696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:02:57.724117 6696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:57.724376 6696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:57.724627 6696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:57.730295 6704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:57.730393 6701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:57.730393 6702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:57.730505 6696 server_base.cc:1034] running on GCE node
I20260924 03:02:57.730857 6696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:57.731069 6696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:57.732208 6696 hybrid_clock.cc:648] HybridClock initialized: now 1790218977732194 us; error 27 us; skew 500 ppm
Sep 24 03:02:57 dist-test-slave-wgsb krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218977, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:57.734818 6696 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 03:02:57.735354 6696 webserver.cc:460] Webserver started at http://127.0.98.2:33001/ using document root <none> and password file <none>
I20260924 03:02:57.735677 6696 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:57.735726 6696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:57.735888 6696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:57.737285 6696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/data/instance:
uuid: "a7ccdcca74be438cb9cd448857cbc55a"
format_stamp: "Formatted at 2026-09-24 03:02:57 on dist-test-slave-wgsb"
server_key: "b3b2e41d75f2a5ac18738e745eed9f97"
server_key_iv: "5d1c818783f79d83e3523d2ba2f619e1"
server_key_version: "encryptionkey@0"
I20260924 03:02:57.737772 6696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a7ccdcca74be438cb9cd448857cbc55a"
format_stamp: "Formatted at 2026-09-24 03:02:57 on dist-test-slave-wgsb"
server_key: "b3b2e41d75f2a5ac18738e745eed9f97"
server_key_iv: "5d1c818783f79d83e3523d2ba2f619e1"
server_key_version: "encryptionkey@0"
I20260924 03:02:57.739295 6696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:02:57.740212 6711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:57.740386 6696 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:02:57.740494 6696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "a7ccdcca74be438cb9cd448857cbc55a"
format_stamp: "Formatted at 2026-09-24 03:02:57 on dist-test-slave-wgsb"
server_key: "b3b2e41d75f2a5ac18738e745eed9f97"
server_key_iv: "5d1c818783f79d83e3523d2ba2f619e1"
server_key_version: "encryptionkey@0"
I20260924 03:02:57.740582 6696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:57.753786 6696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:57.756039 6696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:57.756217 6696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:57.756474 6696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:57.756810 6696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:57.756868 6696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:57.756917 6696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:57.756945 6696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:57.763293 6696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34985
I20260924 03:02:57.763365 6824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34985 every 8 connection(s)
I20260924 03:02:57.763809 6696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:02:57.765440 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 6696
I20260924 03:02:57.765548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:02:57.765723 392 external_mini_cluster.cc:1428] Setting key 9998ce375fd88f863259a45e74c7b5bd
Sep 24 03:02:57 dist-test-slave-wgsb krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218977, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:02:57.774600 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:57.764581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60231 (local address 127.0.98.62:36379)
0924 03:02:57.764668 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:57.764670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:57.764934 (+ 264us) server_negotiation.cc:407] Connection header received
0924 03:02:57.765523 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:57.765525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:57.765537 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:57.765598 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:57.766958 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:57.767544 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:57.768625 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:57.768991 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:57.769074 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:57.771370 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:57.771372 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:57.771373 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:57.771382 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:57.773135 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:57.773529 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:57.773530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:57.773530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:57.773595 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:57.773842 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:57.773843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:57.773843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:57.774011 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:57.774096 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:57.774382 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 03:02:57.774481 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 03:02:57.774622 6827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:57.764548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:60231)
0924 03:02:57.764805 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:57.764844 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:57.765391 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:57.765688 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:57.765691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:57.766598 (+ 907us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:57.766857 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:57.766862 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:57.767689 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:57.767690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:57.768535 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:57.768537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:57.769075 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:57.769076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:57.769174 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:57.769778 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:57.769786 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:57.771269 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:57.773221 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:57.773226 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:57.773228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:57.773448 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:57.773676 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:57.773677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:57.773677 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:57.773772 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:57.774091 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:57.774095 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:57.774220 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 03:02:57.774321 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 03:02:57.774456 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 03:02:57.775179 6825 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36379
I20260924 03:02:57.775303 6825 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:57.775504 6825 heartbeater.cc:511] Master 127.0.98.62:36379 requested a full tablet report, sending...
I20260924 03:02:57.775898 6500 ts_manager.cc:194] Registered new tserver with Master: a7ccdcca74be438cb9cd448857cbc55a (127.0.98.2:34985)
I20260924 03:02:57.776367 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 03:02:57.795421 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36379
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:57.877663 6830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:57.877931 6830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:57.877974 6830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:57.879139 6830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:57.879210 6830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:57.879256 6830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:02:57.880851 6830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:57.881145 6830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:57.881382 6830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:57.886901 6838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:57.886904 6836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:57.886941 6835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:57.887243 6830 server_base.cc:1034] running on GCE node
I20260924 03:02:57.887415 6830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:57.887667 6830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:57.888835 6830 hybrid_clock.cc:648] HybridClock initialized: now 1790218977888819 us; error 29 us; skew 500 ppm
Sep 24 03:02:57 dist-test-slave-wgsb krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218977, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:57.891705 6830 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 03:02:57.892294 6830 webserver.cc:460] Webserver started at http://127.0.98.3:37199/ using document root <none> and password file <none>
I20260924 03:02:57.892537 6830 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:57.892582 6830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:57.892730 6830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:57.893865 6830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/data/instance:
uuid: "b5a56c6090d644fb8935dc68bbaeedd8"
format_stamp: "Formatted at 2026-09-24 03:02:57 on dist-test-slave-wgsb"
server_key: "f431be25d1796b692dfb4e9877e279d2"
server_key_iv: "e2e80e3c0073c05552abd2c5eab6e06f"
server_key_version: "encryptionkey@0"
I20260924 03:02:57.894240 6830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b5a56c6090d644fb8935dc68bbaeedd8"
format_stamp: "Formatted at 2026-09-24 03:02:57 on dist-test-slave-wgsb"
server_key: "f431be25d1796b692dfb4e9877e279d2"
server_key_iv: "e2e80e3c0073c05552abd2c5eab6e06f"
server_key_version: "encryptionkey@0"
I20260924 03:02:57.895678 6830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 03:02:57.896490 6845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:57.896654 6830 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:02:57.896732 6830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "b5a56c6090d644fb8935dc68bbaeedd8"
format_stamp: "Formatted at 2026-09-24 03:02:57 on dist-test-slave-wgsb"
server_key: "f431be25d1796b692dfb4e9877e279d2"
server_key_iv: "e2e80e3c0073c05552abd2c5eab6e06f"
server_key_version: "encryptionkey@0"
I20260924 03:02:57.896886 6830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:57.914506 6830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:57.916515 6830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:57.916676 6830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:57.916917 6830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:57.917212 6830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:57.917244 6830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:57.917287 6830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:57.917315 6830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:57.923146 6830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43013
I20260924 03:02:57.923202 6958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43013 every 8 connection(s)
I20260924 03:02:57.923545 6830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 03:02:57 dist-test-slave-wgsb krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218977, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:02:57.930644 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 6830
I20260924 03:02:57.930769 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:02:57.930972 392 external_mini_cluster.cc:1428] Setting key de1b940ffb53414307d164b25dc853f8
I20260924 03:02:57.933437 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:57.924251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46971 (local address 127.0.98.62:36379)
0924 03:02:57.924299 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:57.924301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:57.924550 (+ 249us) server_negotiation.cc:407] Connection header received
0924 03:02:57.925034 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:57.925035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:57.925046 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:57.925110 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:57.926086 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:57.926634 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:57.927738 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:57.928052 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:57.928145 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:57.930414 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:57.930415 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:57.930415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:57.930452 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:57.932158 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:57.932524 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:57.932524 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:57.932525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:57.932574 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:57.932807 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:57.932808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:57.932808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:57.932938 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:57.933002 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:57.933307 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 03:02:57.933374 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 03:02:57.933662 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:57.924224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:46971)
0924 03:02:57.924380 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:57.924464 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:57.924920 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:57.925182 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:57.925185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:57.925772 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:57.925966 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:57.925971 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:57.926748 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:57.926749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:57.927577 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:57.927579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:57.928150 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:57.928151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:57.928254 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:57.928787 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:57.928795 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:57.930318 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:57.932222 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:57.932227 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:57.932229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:57.932461 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:57.932631 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:57.932632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:57.932632 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:57.932725 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:57.933042 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:57.933047 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:57.933161 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 03:02:57.933249 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 03:02:57.933366 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260924 03:02:57.934158 6959 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36379
I20260924 03:02:57.934240 6959 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:57.934435 6959 heartbeater.cc:511] Master 127.0.98.62:36379 requested a full tablet report, sending...
I20260924 03:02:57.934794 6500 ts_manager.cc:194] Registered new tserver with Master: b5a56c6090d644fb8935dc68bbaeedd8 (127.0.98.3:43013)
I20260924 03:02:57.935192 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46971
I20260924 03:02:57.942925 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260924 03:02:57.971197 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45601
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36379,127.0.98.61:45601
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:58.053500 6964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:58.053824 6964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:58.053866 6964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:58.055101 6964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:02:58.055178 6964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:58.055202 6964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:58.055226 6964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:02:58.055256 6964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:02:58.056895 6964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36379,127.0.98.61:45601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
W20260924 03:02:58.057045 6964 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:36379,127.0.98.61:45601'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 03:02:58.057214 6964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:58.057454 6964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:58.063078 6972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:58.063239 6970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:58.063081 6969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:58.063913 6964 server_base.cc:1034] running on GCE node
I20260924 03:02:58.064208 6964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:58.064517 6964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:58.065663 6964 hybrid_clock.cc:648] HybridClock initialized: now 1790218978065654 us; error 26 us; skew 500 ppm
Sep 24 03:02:58 dist-test-slave-wgsb krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218978, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:58.068222 6964 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 03:02:58.068707 6964 webserver.cc:460] Webserver started at http://127.0.98.61:44425/ using document root <none> and password file <none>
I20260924 03:02:58.068974 6964 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:58.069021 6964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:58.069175 6964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:58.070264 6964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-1/data/instance:
uuid: "065ca39885b54be2bd9fe99c4d2e3fb6"
format_stamp: "Formatted at 2026-09-24 03:02:58 on dist-test-slave-wgsb"
server_key: "17dd3daf5398b62dd77e41e76d920654"
server_key_iv: "f7e1af5510c35e9636e9cf40f908e13c"
server_key_version: "encryptionkey@0"
I20260924 03:02:58.070626 6964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-1/wal/instance:
uuid: "065ca39885b54be2bd9fe99c4d2e3fb6"
format_stamp: "Formatted at 2026-09-24 03:02:58 on dist-test-slave-wgsb"
server_key: "17dd3daf5398b62dd77e41e76d920654"
server_key_iv: "f7e1af5510c35e9636e9cf40f908e13c"
server_key_version: "encryptionkey@0"
I20260924 03:02:58.072062 6964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:02:58.072774 6979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:58.072978 6964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:02:58.073066 6964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-1/wal
uuid: "065ca39885b54be2bd9fe99c4d2e3fb6"
format_stamp: "Formatted at 2026-09-24 03:02:58 on dist-test-slave-wgsb"
server_key: "17dd3daf5398b62dd77e41e76d920654"
server_key_iv: "f7e1af5510c35e9636e9cf40f908e13c"
server_key_version: "encryptionkey@0"
I20260924 03:02:58.073137 6964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:58.106510 6964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:58.108669 6964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:58.108842 6964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:58.112958 6964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45601
I20260924 03:02:58.112984 7031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45601 every 8 connection(s)
I20260924 03:02:58.113375 6964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-1/data/info.pb
I20260924 03:02:58.114463 7032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:58.116076 7032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:02:58.116413 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 6964
Sep 24 03:02:58 dist-test-slave-wgsb krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218978, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 03:02:58 dist-test-slave-wgsb krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 03:02:58.127241 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.116922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54178 (local address 127.0.98.62:36379)
0924 03:02:58.116971 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.116973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.117276 (+ 303us) server_negotiation.cc:407] Connection header received
0924 03:02:58.117997 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.117998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.118012 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.118086 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.118716 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.119468 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.120437 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.120625 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.120713 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.123294 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.123296 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.123296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.123304 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.125054 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.125824 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.125826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.125826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.125878 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.126224 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.126226 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.126226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.126459 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.126530 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.126932 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.127020 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 03:02:58.127342 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.116968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.0.1:54178)
0924 03:02:58.117148 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:58.117188 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:58.117690 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:58.118150 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:58.118152 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:58.118441 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:58.118626 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.118631 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.119567 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.119567 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.120329 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.120331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.120719 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.120720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.120836 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.121430 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:58.121441 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:58.123203 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:58.125154 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.125160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.125162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.125602 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.125956 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.125957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.125957 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.126127 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.126545 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:58.126551 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:58.126670 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 03:02:58.126884 (+ 214us) client_negotiation.cc:240] Negotiation successful
0924 03:02:58.127047 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
I20260924 03:02:58.127621 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.116963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51794 (local address 127.0.98.61:45601)
0924 03:02:58.117158 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.117167 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.117198 (+ 31us) server_negotiation.cc:407] Connection header received
0924 03:02:58.117687 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.117696 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.117722 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.117812 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.118541 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.119804 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.120521 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.120839 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.120920 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.123779 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.123785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.123785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.123792 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.125590 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.126127 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.126130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.126130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.126214 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.126479 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.126480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.126481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.126672 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.126769 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.127151 (+ 382us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.127289 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":108,"threads_started":1}
I20260924 03:02:58.128400 7032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:02:58 dist-test-slave-wgsb krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218978, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 03:02:58.134473 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.128681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45601 (local address 127.0.0.1:51808)
0924 03:02:58.128717 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:58.128726 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:58.128762 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:58.128872 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:58.128872 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:58.129002 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:58.129043 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.129044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.129655 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.129656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.130133 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.130134 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.130512 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.130513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.130573 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.130991 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:58.130996 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:58.132170 (+ 1174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:58.133523 (+ 1353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.133524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.133524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.133682 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.133882 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.133883 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.133883 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.133935 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.134173 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:58.134174 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:58.134208 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 03:02:58.134274 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 03:02:58.134362 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260924 03:02:58.134490 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.128734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51808 (local address 127.0.98.61:45601)
0924 03:02:58.128765 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.128767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.128774 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:02:58.128813 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.128813 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.128822 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.128862 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.129115 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.129578 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.130245 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.130430 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.130513 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.132267 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.132268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.132269 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.132275 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.133415 (+ 1140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.133763 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.133763 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.133764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.133818 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.134005 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.134006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.134006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.134106 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.134181 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.134334 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.134407 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":17}
I20260924 03:02:58.136198 7032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6: Bootstrap starting.
I20260924 03:02:58.137364 7032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:58.137607 7032 log.cc:824] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:58.138310 7032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6: No bootstrap required, opened a new log
I20260924 03:02:58.139848 7032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } }
I20260924 03:02:58.139983 7032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:58.140007 7032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 065ca39885b54be2bd9fe99c4d2e3fb6, State: Initialized, Role: FOLLOWER
I20260924 03:02:58.140137 7032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } }
I20260924 03:02:58.140420 7038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } } }
I20260924 03:02:58.140468 7032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:02:58.140523 7038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 03:02:58.142351 7049 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:02:58.142616 7049 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:02:58.144078 6964 master_runner.cc:428] Detected that this master 065ca39885b54be2bd9fe99c4d2e3fb6 is joining an existing cluster
I20260924 03:02:58.144147 6964 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:45601
I20260924 03:02:58.151717 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.145305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45601 (local address 127.0.0.1:51810)
0924 03:02:58.145481 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:58.145490 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:58.145545 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:58.145659 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:58.145660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:58.145968 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:58.146050 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.146053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.146940 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.146940 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.147784 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.147787 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.148198 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.148199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.148261 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.148626 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:58.148631 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:58.148939 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:58.150615 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.150616 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.150617 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.150811 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.151032 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.151032 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.151033 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.151087 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.151343 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:58.151344 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:58.151380 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 03:02:58.151461 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 03:02:58.151560 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":1,"thread_start_us":70,"threads_started":1}
I20260924 03:02:58.151718 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.145354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51810 (local address 127.0.98.61:45601)
0924 03:02:58.145428 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.145430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.145533 (+ 103us) server_negotiation.cc:407] Connection header received
0924 03:02:58.145591 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.145592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.145601 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.145653 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.146146 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.146854 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.147863 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.148075 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.148153 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.149028 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.149030 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.149030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.149036 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.150516 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.150888 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.150889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.150889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.150953 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.151161 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.151161 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.151162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.151267 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.151340 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.151483 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.151631 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":58}
I20260924 03:02:58.151902 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.145163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54186 (local address 127.0.98.62:36379)
0924 03:02:58.145203 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.145205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.145502 (+ 297us) server_negotiation.cc:407] Connection header received
0924 03:02:58.145595 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.145596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.145607 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.145639 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.146062 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.146584 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.147343 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.147549 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.147664 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.148771 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.148772 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.148773 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.148780 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.150562 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.151005 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.151006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.151007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.151064 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.151289 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.151291 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.151291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.151430 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.151475 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.151729 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.151807 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:02:58.152503 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.145094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.0.1:54186)
0924 03:02:58.145419 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:58.145429 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:58.145515 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:58.145700 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:58.145701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:58.145877 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:58.145981 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.145983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.146677 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.146678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.147250 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.147252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.147676 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.147677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.147741 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.148240 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:58.148245 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:58.148670 (+ 425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:58.150654 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.150655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.150655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.150930 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.151135 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.151136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.151136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.151203 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.151523 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:58.151523 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:58.151576 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 03:02:58.152288 (+ 712us) client_negotiation.cc:240] Negotiation successful
0924 03:02:58.152398 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":2,"thread_start_us":92,"threads_started":1}
Sep 24 03:02:58 dist-test-slave-wgsb krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218977, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 03:02:58.162036 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.154678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45601 (local address 127.0.0.1:51820)
0924 03:02:58.154843 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:58.154880 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:58.154954 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:58.155085 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:58.155089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:58.155441 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:58.155507 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.155509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.156168 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.156169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.157108 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.157111 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.157459 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.157460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.157539 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.157942 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:58.157946 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:58.159396 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:58.160997 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.161003 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.161005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.161194 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.161439 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.161439 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.161439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.161491 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.161739 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:58.161744 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:58.161787 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 03:02:58.161851 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 03:02:58.161965 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":66,"threads_started":1}
I20260924 03:02:58.162060 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.154728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51820 (local address 127.0.98.61:45601)
0924 03:02:58.154761 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.154763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.154954 (+ 191us) server_negotiation.cc:407] Connection header received
0924 03:02:58.155024 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.155024 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.155032 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.155079 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.155581 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.156089 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.157189 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.157386 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.157443 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.159476 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.159477 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.159477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.159484 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.160917 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.161264 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.161265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.161265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.161320 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.161550 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.161551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.161551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.161676 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.161745 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.161896 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.161967 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:02:58.162935 6500 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:45601: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 45601
}
attrs {
promote: true
}
}
dest_uuid: "105f4753f35f455c9ebfdf86ba7c065c"
cas_config_opid_index: -1
I20260924 03:02:58.163249 6500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: true } }
I20260924 03:02:58.163745 7059 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 065ca39885b54be2bd9fe99c4d2e3fb6 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: true } } }
I20260924 03:02:58.164042 7006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:58.164832 7059 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:45601
I20260924 03:02:58.165031 7006 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 1 FOLLOWER]: Refusing update from remote peer 105f4753f35f455c9ebfdf86ba7c065c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 03:02:58.165050 6964 master.cc:537] Master@127.0.98.61:45601 shutting down...
I20260924 03:02:58.165069 7060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "105f4753f35f455c9ebfdf86ba7c065c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: true } } }
I20260924 03:02:58.165411 7059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [LEADER]: Connected to new peer: Peer: permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 03:02:58.166103 6486 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 03:02:58.166556 6486 consensus_peers.cc:487] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c -> Peer 065ca39885b54be2bd9fe99c4d2e3fb6 (127.0.98.61:45601): Couldn't send request to peer 065ca39885b54be2bd9fe99c4d2e3fb6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 03:02:58.166574 7060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:58.167516 7038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:02:58.167721 7038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } }
I20260924 03:02:58.167918 7038 leader_election.cc:288] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 105f4753f35f455c9ebfdf86ba7c065c (127.0.98.62:36379)
I20260924 03:02:58.168076 6964 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:02:58.168100 6510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "105f4753f35f455c9ebfdf86ba7c065c" is_pre_election: true
I20260924 03:02:58.168164 6964 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:02:58.168211 6964 tablet_replica.cc:330] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6: stopping tablet replica
I20260924 03:02:58.168401 6982 leader_election.cc:302] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 065ca39885b54be2bd9fe99c4d2e3fb6; no voters: 105f4753f35f455c9ebfdf86ba7c065c
W20260924 03:02:58.168488 6982 raft_consensus.cc:2670] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260924 03:02:58.180464 6964 master.cc:552] Master@127.0.98.61:45601 shutdown complete.
I20260924 03:02:58.180511 6964 master_runner.cc:305] Clearing existing system tablet
I20260924 03:02:58.180657 6964 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:58.180697 6964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:58.181298 6964 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:02:58.181806 7064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:58.181926 6964 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:58.181982 6964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:02:58.182006 6964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-1/wal
uuid: "065ca39885b54be2bd9fe99c4d2e3fb6"
format_stamp: "Formatted at 2026-09-24 03:02:58 on dist-test-slave-wgsb"
server_key: "17dd3daf5398b62dd77e41e76d920654"
server_key_iv: "f7e1af5510c35e9636e9cf40f908e13c"
server_key_version: "encryptionkey@0"
I20260924 03:02:58.182623 6964 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:02:58.184124 6964 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 03:02:58.184207 6964 log.cc:1196] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 03:02:58.184407 6964 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6: Deleting consensus metadata
I20260924 03:02:58.184554 6964 master_runner.cc:315] Copying system tablet from 127.0.98.62:36379
I20260924 03:02:58.185180 6964 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:36379
I20260924 03:02:58.191708 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.185565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54190 (local address 127.0.98.62:36379)
0924 03:02:58.185628 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.185631 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.185748 (+ 117us) server_negotiation.cc:407] Connection header received
0924 03:02:58.185780 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.185781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.185792 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.185868 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.186199 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.186989 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.187798 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.188104 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.188178 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.189116 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.189117 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.189117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.189124 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.190646 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.190966 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.190967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.190968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.191029 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.191194 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.191194 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.191195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.191323 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.191400 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.191495 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.191569 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 03:02:58.191720 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.185515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.0.1:54190)
0924 03:02:58.185679 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:58.185688 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:58.185740 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:58.185870 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:58.185871 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:58.186047 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:58.186105 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.186107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.187083 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.187084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.187721 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.187723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.188180 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.188181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.188231 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.188708 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:58.188713 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:58.189033 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:58.190725 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.190726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.190726 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.190889 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.191092 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.191093 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.191093 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.191140 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.191385 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:58.191386 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:58.191421 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 03:02:58.191472 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 03:02:58.191545 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":65,"threads_started":1}
I20260924 03:02:58.192065 6520 tablet_copy_service.cc:140] P 105f4753f35f455c9ebfdf86ba7c065c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 065ca39885b54be2bd9fe99c4d2e3fb6 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:54190)
I20260924 03:02:58.192140 6520 tablet_copy_service.cc:161] P 105f4753f35f455c9ebfdf86ba7c065c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 065ca39885b54be2bd9fe99c4d2e3fb6 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:54190: session id = 065ca39885b54be2bd9fe99c4d2e3fb6-00000000000000000000000000000000
I20260924 03:02:58.192742 6520 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 03:02:58.193297 6964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 03:02:58.194195 6964 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6: tablet copy: Starting download of 0 data blocks...
I20260924 03:02:58.194342 6964 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6: tablet copy: Starting download of 1 WAL segments...
I20260924 03:02:58.195142 6964 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 03:02:58.195976 6520 tablet_copy_service.cc:342] P 105f4753f35f455c9ebfdf86ba7c065c: Request end of tablet copy session 065ca39885b54be2bd9fe99c4d2e3fb6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:54190
I20260924 03:02:58.196049 6520 tablet_copy_service.cc:433] P 105f4753f35f455c9ebfdf86ba7c065c: ending tablet copy session 065ca39885b54be2bd9fe99c4d2e3fb6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 065ca39885b54be2bd9fe99c4d2e3fb6
I20260924 03:02:58.198498 6964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:58.199785 7074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:58.200003 7072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:58.200055 6964 server_base.cc:1034] running on GCE node
W20260924 03:02:58.200052 7071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:58.200243 6964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:58.200399 6964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:58.201535 6964 hybrid_clock.cc:648] HybridClock initialized: now 1790218978201527 us; error 24 us; skew 500 ppm
Sep 24 03:02:58 dist-test-slave-wgsb krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218978, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:58.202792 6964 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 03:02:58.203079 6964 webserver.cc:460] Webserver started at http://127.0.98.61:35053/ using document root <none> and password file <none>
I20260924 03:02:58.203191 6964 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:58.203217 6964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:58.203816 6964 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:02:58.204306 7081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:58.204433 6964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:02:58.204481 6964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-1/wal
uuid: "065ca39885b54be2bd9fe99c4d2e3fb6"
format_stamp: "Formatted at 2026-09-24 03:02:58 on dist-test-slave-wgsb"
server_key: "17dd3daf5398b62dd77e41e76d920654"
server_key_iv: "f7e1af5510c35e9636e9cf40f908e13c"
server_key_version: "encryptionkey@0"
I20260924 03:02:58.204540 6964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:58.211248 6964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:58.212833 6964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:58.215864 6964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45601
I20260924 03:02:58.216192 7133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45601 every 8 connection(s)
I20260924 03:02:58.216298 6964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-1/data/info.pb
I20260924 03:02:58.217028 7134 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:02:58.217403 7134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6: Bootstrap starting.
I20260924 03:02:58.219719 7134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:02:58.219949 7134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6: Bootstrap complete.
I20260924 03:02:58.220160 7134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: true } }
I20260924 03:02:58.220248 7134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 065ca39885b54be2bd9fe99c4d2e3fb6, State: Initialized, Role: LEARNER
I20260924 03:02:58.220299 7134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: true } }
I20260924 03:02:58.220499 7134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:02:58.220486 7137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: true } } }
I20260924 03:02:58.220566 7137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [sys.catalog]: This master's current role is: LEARNER
I20260924 03:02:58.221959 7148 catalog_manager.cc:1259] Loaded cluster ID: 66173977c15d45668f6c482930c04bf9
I20260924 03:02:58.222023 7148 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6: loading cluster ID for follower catalog manager: success
I20260924 03:02:58.223097 7148 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6: acquiring CA information for follower catalog manager: success
I20260924 03:02:58.223378 7148 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:02:58.235786 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.229274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51844 (local address 127.0.98.61:45601)
0924 03:02:58.229429 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.229430 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.229438 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:02:58.229459 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.229460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.229469 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.229542 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.230112 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.230794 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.231530 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.231874 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.231941 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.233187 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.233189 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.233189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.233197 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.234615 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.234971 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.234972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.234973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.235035 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.235235 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.235236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.235236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.235348 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.235468 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.235563 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.235665 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260924 03:02:58.262941 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40835
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36379,127.0.98.61:45601,127.0.98.60:40835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:02:58.348438 7152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:58.348660 7152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:58.348685 7152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:58.350078 7152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:02:58.350126 7152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:58.350143 7152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:58.350157 7152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:02:58.350169 7152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:02:58.351738 7152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36379,127.0.98.61:45601,127.0.98.60:40835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:58.351949 7152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:58.352126 7152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:58.356827 7157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:58.356894 7158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:58.357028 7152 server_base.cc:1034] running on GCE node
W20260924 03:02:58.356834 7160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:58.357316 7152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:58.357620 7152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:58.358783 7152 hybrid_clock.cc:648] HybridClock initialized: now 1790218978358771 us; error 29 us; skew 500 ppm
Sep 24 03:02:58 dist-test-slave-wgsb krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218978, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:58.361270 7152 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 03:02:58.361646 7152 webserver.cc:460] Webserver started at http://127.0.98.60:35903/ using document root <none> and password file <none>
I20260924 03:02:58.361835 7152 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:58.361876 7152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:58.361963 7152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:02:58.362946 7152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-2/data/instance:
uuid: "6e6eae6748bd4849aa05bad1e22daf60"
format_stamp: "Formatted at 2026-09-24 03:02:58 on dist-test-slave-wgsb"
server_key: "aa2a760977f36de4b4e0c6a0d2e0d66c"
server_key_iv: "7087ad230845cc7ed47d92cfbe787ed7"
server_key_version: "encryptionkey@0"
I20260924 03:02:58.363236 7152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-2/wal/instance:
uuid: "6e6eae6748bd4849aa05bad1e22daf60"
format_stamp: "Formatted at 2026-09-24 03:02:58 on dist-test-slave-wgsb"
server_key: "aa2a760977f36de4b4e0c6a0d2e0d66c"
server_key_iv: "7087ad230845cc7ed47d92cfbe787ed7"
server_key_version: "encryptionkey@0"
I20260924 03:02:58.364605 7152 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:02:58.365314 7167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:58.365537 7152 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:58.365625 7152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-2/wal
uuid: "6e6eae6748bd4849aa05bad1e22daf60"
format_stamp: "Formatted at 2026-09-24 03:02:58 on dist-test-slave-wgsb"
server_key: "aa2a760977f36de4b4e0c6a0d2e0d66c"
server_key_iv: "7087ad230845cc7ed47d92cfbe787ed7"
server_key_version: "encryptionkey@0"
I20260924 03:02:58.365682 7152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:58.383807 7152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:58.386225 7152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:58.386339 7152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:58.390645 7152 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40835
I20260924 03:02:58.390702 7219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40835 every 8 connection(s)
I20260924 03:02:58.391060 7152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-2/data/info.pb
I20260924 03:02:58.392198 7220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:58.393925 7220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:02:58.398403 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 7152
Sep 24 03:02:58 dist-test-slave-wgsb krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218978, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:02:58.404074 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.394888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54202 (local address 127.0.98.62:36379)
0924 03:02:58.394949 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.394952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.395073 (+ 121us) server_negotiation.cc:407] Connection header received
0924 03:02:58.395581 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.395582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.395614 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.395686 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.396327 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.396915 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.397969 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.398148 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.398198 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.400932 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.400934 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.400934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.400943 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.402598 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.403015 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.403016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.403017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.403082 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.403390 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.403392 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.403392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.403517 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.403596 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.403895 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.403981 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 03:02:58.404278 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.394735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.0.1:54202)
0924 03:02:58.394933 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:58.394978 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:58.395433 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:58.395740 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:58.395798 (+ 58us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:58.396035 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:58.396222 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.396226 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.397007 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.397008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.397872 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.397874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.398235 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.398236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.398404 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.398988 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:58.398996 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:58.400823 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:58.402677 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.402683 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.402685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.402902 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.403152 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.403153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.403153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.403268 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.403587 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:58.403615 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:58.403758 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 03:02:58.403872 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 03:02:58.404039 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":106,"threads_started":1}
I20260924 03:02:58.404875 7220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:02:58 dist-test-slave-wgsb krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 03:02:58 dist-test-slave-wgsb krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218978, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 03:02:58.408883 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.399105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40136 (local address 127.0.98.60:40835)
0924 03:02:58.399321 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.399325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.399343 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:02:58.399361 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.399363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.399379 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.399643 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.400323 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.401380 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.402366 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.402880 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.402969 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.405881 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.405887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.405887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.405893 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.407405 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.408028 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.408031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.408031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.408090 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.408311 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.408312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.408313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.408450 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.408609 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.408687 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.408759 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":91,"threads_started":1}
I20260924 03:02:58.411382 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.405174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45601 (local address 127.0.0.1:51854)
0924 03:02:58.405326 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:58.405336 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:58.405410 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:58.405616 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:58.405617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:58.405743 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:58.405786 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.405787 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.406456 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.406457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.406954 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.406955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.407339 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.407340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.407409 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.407764 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:58.407768 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:58.408936 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:58.410585 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.410586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.410586 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.410740 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.410962 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.410963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.410963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.411011 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.411180 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:58.411181 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:58.411212 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 03:02:58.411261 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 03:02:58.411325 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260924 03:02:58.411398 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.405244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51854 (local address 127.0.98.61:45601)
0924 03:02:58.405298 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.405300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.405393 (+ 93us) server_negotiation.cc:407] Connection header received
0924 03:02:58.405512 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.405514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.405531 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.405587 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.405861 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.406346 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.407032 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.407253 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.407338 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.409049 (+ 1711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.409050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.409051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.409058 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.410513 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.410856 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.410856 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.410857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.410904 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.411049 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.411050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.411050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.411135 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.411178 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.411278 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.411339 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 03:02:58.411913 7220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40835 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:02:58 dist-test-slave-wgsb krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218978, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 03:02:58.418331 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.412145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40835 (local address 127.0.0.1:40142)
0924 03:02:58.412230 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:58.412239 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:58.412282 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:58.412367 (+ 85us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:58.412368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:58.412490 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:58.412517 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.412518 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.413131 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.413131 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.413591 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.413592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.414039 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.414040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.414083 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.414461 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:58.414465 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:58.415680 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:58.417370 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.417371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.417371 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.417536 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.417805 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.417806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.417806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.417857 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.418120 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:58.418121 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:58.418152 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 03:02:58.418202 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 03:02:58.418273 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 03:02:58.418430 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.412196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40142 (local address 127.0.98.60:40835)
0924 03:02:58.412225 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.412227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.412271 (+ 44us) server_negotiation.cc:407] Connection header received
0924 03:02:58.412316 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.412316 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.412324 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.412366 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.412563 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.413069 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.413688 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.413947 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.414019 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.415806 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.415808 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.415808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.415814 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.417289 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.417657 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.417658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.417659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.417733 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.417922 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.417923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.417923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.418051 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.418125 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.418232 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.418336 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":14}
I20260924 03:02:58.420495 7220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60: Bootstrap starting.
I20260924 03:02:58.421371 7220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:58.421660 7220 log.cc:824] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:58.422407 7220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60: No bootstrap required, opened a new log
I20260924 03:02:58.424408 7220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40835 } }
I20260924 03:02:58.424551 7220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:58.424575 7220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e6eae6748bd4849aa05bad1e22daf60, State: Initialized, Role: FOLLOWER
I20260924 03:02:58.424669 7220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40835 } }
I20260924 03:02:58.424940 7225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40835 } } }
I20260924 03:02:58.424988 7220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:02:58.425040 7225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 03:02:58.426617 7236 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:02:58.426674 7236 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:02:58.429104 7152 master_runner.cc:428] Detected that this master 6e6eae6748bd4849aa05bad1e22daf60 is joining an existing cluster
I20260924 03:02:58.429157 7152 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:40835
I20260924 03:02:58.437613 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.430279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40154 (local address 127.0.98.60:40835)
0924 03:02:58.430323 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.430325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.431123 (+ 798us) server_negotiation.cc:407] Connection header received
0924 03:02:58.431192 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.431193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.431205 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.431259 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.431751 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.432503 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.433541 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.433815 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.433850 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.435037 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.435039 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.435039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.435046 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.436452 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.436806 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.436807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.436807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.436862 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.437051 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.437052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.437053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.437159 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.437227 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.437378 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.437526 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":22}
I20260924 03:02:58.437999 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.430171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40835 (local address 127.0.0.1:40154)
0924 03:02:58.430595 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:58.430607 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:58.431121 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:58.431301 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:58.431302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:58.431573 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:58.431670 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.431672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.432581 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.432582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.433459 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.433461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.434059 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.434060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.434120 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.434612 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:58.434616 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:58.434950 (+ 334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:58.436548 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.436548 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.436549 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.436725 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.436933 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.436934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.436934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.436986 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.437227 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:58.437228 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:58.437280 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 03:02:58.437359 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 03:02:58.437476 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":3,"thread_start_us":449,"threads_started":1}
I20260924 03:02:58.456079 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.430658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51864 (local address 127.0.98.61:45601)
0924 03:02:58.430702 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.430704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.431244 (+ 540us) server_negotiation.cc:407] Connection header received
0924 03:02:58.431298 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.431299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.431311 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.431369 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.431934 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.432636 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.433747 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.434023 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.434067 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.453227 (+ 19160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.453229 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.453229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.453237 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.454731 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.455099 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.455100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.455101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.455159 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.455388 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.455389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.455390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.455546 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.455676 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.455881 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.455975 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:02:58.456089 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.430404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45601 (local address 127.0.0.1:51864)
0924 03:02:58.431188 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:58.431196 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:58.431245 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:58.431420 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:58.431421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:58.431756 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:58.431829 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.431831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.432748 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.432749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.433635 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.433638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.434148 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.434149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.434222 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.452550 (+ 18328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:58.452555 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:58.453096 (+ 541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:58.454841 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.454842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.454843 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.455026 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.455227 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.455228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.455229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.455308 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.455701 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:58.455702 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:58.455752 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 03:02:58.455846 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 03:02:58.455964 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":27,"thread_start_us":111,"threads_started":1}
I20260924 03:02:58.456413 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.430360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54216 (local address 127.0.98.62:36379)
0924 03:02:58.430616 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.430618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.431134 (+ 516us) server_negotiation.cc:407] Connection header received
0924 03:02:58.431378 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.431380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.431390 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.431420 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.431930 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.432652 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.433716 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.433984 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.434023 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.453265 (+ 19242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.453267 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.453267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.453277 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.455012 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.455536 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.455537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.455538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.455612 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.455800 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.455801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.455801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.455947 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.456011 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.456198 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.456293 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226}
I20260924 03:02:58.456700 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.430171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.0.1:54216)
0924 03:02:58.430720 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:58.430730 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:58.431305 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:58.431503 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:58.431503 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:58.431794 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:58.431845 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.431847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.432732 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.432733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.433637 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.433638 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.434092 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.434092 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.434145 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.452264 (+ 18119us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:58.452272 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:58.453114 (+ 842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:58.455248 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.455249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.455250 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.455464 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.455675 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.455676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.455676 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.455727 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.456038 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:58.456039 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:58.456083 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 03:02:58.456497 (+ 414us) client_negotiation.cc:240] Negotiation successful
0924 03:02:58.456605 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":41,"thread_start_us":96,"threads_started":1}
I20260924 03:02:58.460155 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 6562
Sep 24 03:02:58 dist-test-slave-wgsb krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218977, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 03:02:58.466068 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.458621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40835 (local address 127.0.0.1:40162)
0924 03:02:58.458670 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:58.458679 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:58.458712 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:58.458838 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:58.458839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:58.458993 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:58.459040 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.459042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.459797 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.459798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.460634 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.460637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.461155 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.461156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.461233 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.461633 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:58.461638 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:58.463180 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:58.464962 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.464963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.464963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.465168 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.465378 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.465379 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.465379 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.465448 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.465744 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:58.465745 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:58.465794 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 03:02:58.465865 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 03:02:58.465966 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 03:02:58.466096 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33289
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36379
--webserver_port=35349
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:36379,127.0.98.61:45601
--tserver_master_addrs=127.0.98.62:36379,127.0.98.61:45601,127.0.98.60:40835 with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:02:58.466447 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.458691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40162 (local address 127.0.98.60:40835)
0924 03:02:58.458728 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.458730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.458737 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:02:58.458769 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.458770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.458784 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.458824 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.459103 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.459712 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.460708 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.461036 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.461103 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.463273 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.463274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.463275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.463281 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.464869 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.465243 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.465244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.465245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.465305 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.465525 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.465526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.465526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.465671 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.465739 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.466267 (+ 528us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.466356 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:02:58.467208 6500 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:40835: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 40835
}
attrs {
promote: true
}
}
dest_uuid: "105f4753f35f455c9ebfdf86ba7c065c"
cas_config_opid_index: 5
I20260924 03:02:58.467504 6500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: true } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: true } }
I20260924 03:02:58.467903 7060 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6e6eae6748bd4849aa05bad1e22daf60 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: true } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: true } } }
I20260924 03:02:58.469044 7060 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:40835
I20260924 03:02:58.469122 7060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "105f4753f35f455c9ebfdf86ba7c065c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: true } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: true } } }
I20260924 03:02:58.469197 7060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:58.469412 7194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:58.470546 7194 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 1 FOLLOWER]: Refusing update from remote peer 105f4753f35f455c9ebfdf86ba7c065c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 03:02:58.471114 7059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:02:58.471649 7152 master.cc:537] Master@127.0.98.60:40835 shutting down...
I20260924 03:02:58.472863 7194 raft_consensus.cc:858] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40835 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: true } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: true } }
W20260924 03:02:58.472935 7194 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40835 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: true } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: true } }
I20260924 03:02:58.473454 7194 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 065ca39885b54be2bd9fe99c4d2e3fb6 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: true } } }
I20260924 03:02:58.474749 7225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 105f4753f35f455c9ebfdf86ba7c065c. Latest consensus state: current_term: 1 leader_uuid: "105f4753f35f455c9ebfdf86ba7c065c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: true } } }
I20260924 03:02:58.474843 7225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260924 03:02:58.474879 7225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "105f4753f35f455c9ebfdf86ba7c065c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: true } } }
I20260924 03:02:58.474920 7225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260924 03:02:58.474947 7225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "105f4753f35f455c9ebfdf86ba7c065c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: true } } }
I20260924 03:02:58.474984 7225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260924 03:02:58.475370 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.467978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45601 (local address 127.0.0.1:51866)
0924 03:02:58.468278 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:58.468289 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:58.468366 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:58.468573 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:58.468574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:58.468575 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:58.468614 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.468616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.469644 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.469645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.470905 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.470907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.475077 (+ 4170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.475078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.475155 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.475183 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 03:02:58.475240 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 03:02:58.475286 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286}
I20260924 03:02:58.475425 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.468063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51866 (local address 127.0.98.61:45601)
0924 03:02:58.468102 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.468104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.468337 (+ 233us) server_negotiation.cc:407] Connection header received
0924 03:02:58.468440 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.468441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.468462 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.468502 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:58.468685 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.469488 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.471009 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.474574 (+ 3565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.474631 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.474712 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.475263 (+ 551us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.475322 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:02:58.475683 7108 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 1 LEARNER]: Refusing update from remote peer 105f4753f35f455c9ebfdf86ba7c065c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 03:02:58.475878 7059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [LEADER]: Connected to new peer: Peer: permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 03:02:58.476413 7108 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6e6eae6748bd4849aa05bad1e22daf60 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: true } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: true } } }
I20260924 03:02:58.477669 7059 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c: attempting to promote NON_VOTER 065ca39885b54be2bd9fe99c4d2e3fb6 to VOTER
I20260924 03:02:58.477691 7137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 105f4753f35f455c9ebfdf86ba7c065c. Latest consensus state: current_term: 1 leader_uuid: "105f4753f35f455c9ebfdf86ba7c065c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: true } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: true } } }
I20260924 03:02:58.477769 7137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [sys.catalog]: This master's current role is: LEARNER
I20260924 03:02:58.477805 7137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "105f4753f35f455c9ebfdf86ba7c065c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: true } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: true } } }
I20260924 03:02:58.477845 7137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [sys.catalog]: This master's current role is: LEARNER
I20260924 03:02:58.477875 7059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: true } }
I20260924 03:02:58.478156 7246 mvcc.cc:205] Tried to move back new op lower bound from 7332736931623288832 to 7332736931542630400. Current Snapshot: MvccSnapshot[applied={T|T < 7332736931552374784}]
W20260924 03:02:58.478780 6487 consensus_peers.cc:487] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c -> Peer 6e6eae6748bd4849aa05bad1e22daf60 (127.0.98.60:40835): Couldn't send request to peer 6e6eae6748bd4849aa05bad1e22daf60. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 03:02:58.478999 7108 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 1 LEARNER]: Refusing update from remote peer 105f4753f35f455c9ebfdf86ba7c065c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 03:02:58.479171 7058 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [LEADER]: Connected to new peer: Peer: permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 03:02:58.479794 7058 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 065ca39885b54be2bd9fe99c4d2e3fb6 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: true } } }
I20260924 03:02:58.480643 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "105f4753f35f455c9ebfdf86ba7c065c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: true } } }
I20260924 03:02:58.480710 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:58.480751 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "105f4753f35f455c9ebfdf86ba7c065c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: true } } }
I20260924 03:02:58.480794 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:58.481123 7108 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 065ca39885b54be2bd9fe99c4d2e3fb6 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: true } } }
I20260924 03:02:58.482084 7251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "105f4753f35f455c9ebfdf86ba7c065c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: true } } }
I20260924 03:02:58.482177 7251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:02:58.484959 7152 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260924 03:02:58.485059 7152 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260924 03:02:58.485088 7152 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60: stopping tablet replica
I20260924 03:02:58.498385 7152 master.cc:552] Master@127.0.98.60:40835 shutdown complete.
I20260924 03:02:58.498440 7152 master_runner.cc:305] Clearing existing system tablet
I20260924 03:02:58.498566 7152 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:58.498611 7152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:58.499317 7152 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:02:58.499991 7256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:58.500144 7152 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:58.500216 7152 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 03:02:58.500244 7152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-2/wal
uuid: "6e6eae6748bd4849aa05bad1e22daf60"
format_stamp: "Formatted at 2026-09-24 03:02:58 on dist-test-slave-wgsb"
server_key: "aa2a760977f36de4b4e0c6a0d2e0d66c"
server_key_iv: "7087ad230845cc7ed47d92cfbe787ed7"
server_key_version: "encryptionkey@0"
I20260924 03:02:58.501080 7152 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:02:58.502774 7152 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 03:02:58.502848 7152 log.cc:1196] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 03:02:58.503088 7152 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60: Deleting consensus metadata
I20260924 03:02:58.503201 7152 master_runner.cc:315] Copying system tablet from 127.0.98.62:36379
I20260924 03:02:58.503804 7152 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:36379
I20260924 03:02:58.512717 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.504187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54226 (local address 127.0.98.62:36379)
0924 03:02:58.504242 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.504245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.505851 (+ 1606us) server_negotiation.cc:407] Connection header received
0924 03:02:58.505895 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.505896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.505908 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.505990 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.506361 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.507156 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.508184 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.508501 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.508576 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.509651 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.509653 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.509653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.509660 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.511350 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.511809 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.511810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.511811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.511873 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.512127 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.512128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.512128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.512261 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.512345 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.512540 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.512622 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 03:02:58.512768 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.504086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.0.1:54226)
0924 03:02:58.505758 (+ 1672us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:58.505772 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:58.505845 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:58.506002 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:58.506003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:58.506201 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:58.506269 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.506271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.507249 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.507250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.508092 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.508095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.508584 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.508585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.508655 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.509172 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:58.509178 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:58.509557 (+ 379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:58.511433 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.511434 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.511434 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.511709 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.511960 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.511961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.511962 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.512043 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.512352 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:58.512353 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:58.512409 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 03:02:58.512496 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 03:02:58.512623 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1649,"thread_start_us":70,"threads_started":1}
I20260924 03:02:58.513425 6520 tablet_copy_service.cc:140] P 105f4753f35f455c9ebfdf86ba7c065c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6e6eae6748bd4849aa05bad1e22daf60 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:54226)
I20260924 03:02:58.513494 6520 tablet_copy_service.cc:161] P 105f4753f35f455c9ebfdf86ba7c065c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6e6eae6748bd4849aa05bad1e22daf60 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:54226: session id = 6e6eae6748bd4849aa05bad1e22daf60-00000000000000000000000000000000
I20260924 03:02:58.514073 6520 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 03:02:58.514515 7152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 03:02:58.515744 7152 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60: tablet copy: Starting download of 0 data blocks...
I20260924 03:02:58.515878 7152 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60: tablet copy: Starting download of 1 WAL segments...
I20260924 03:02:58.516707 7152 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 03:02:58.517598 6520 tablet_copy_service.cc:342] P 105f4753f35f455c9ebfdf86ba7c065c: Request end of tablet copy session 6e6eae6748bd4849aa05bad1e22daf60-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:54226
I20260924 03:02:58.517648 6520 tablet_copy_service.cc:433] P 105f4753f35f455c9ebfdf86ba7c065c: ending tablet copy session 6e6eae6748bd4849aa05bad1e22daf60-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6e6eae6748bd4849aa05bad1e22daf60
I20260924 03:02:58.522540 7152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:58.527930 7264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:58.528671 7263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:58.529259 7266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:58.531095 7152 server_base.cc:1034] running on GCE node
I20260924 03:02:58.531209 7152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:58.531374 7152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:58.532521 7152 hybrid_clock.cc:648] HybridClock initialized: now 1790218978532512 us; error 25 us; skew 500 ppm
Sep 24 03:02:58 dist-test-slave-wgsb krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218978, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:58.534046 7152 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 03:02:58.534308 7152 webserver.cc:460] Webserver started at http://127.0.98.60:35549/ using document root <none> and password file <none>
I20260924 03:02:58.534425 7152 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:58.534461 7152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:58.535221 7152 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:58.535826 7273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:58.535943 7152 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:58.536000 7152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-2/wal
uuid: "6e6eae6748bd4849aa05bad1e22daf60"
format_stamp: "Formatted at 2026-09-24 03:02:58 on dist-test-slave-wgsb"
server_key: "aa2a760977f36de4b4e0c6a0d2e0d66c"
server_key_iv: "7087ad230845cc7ed47d92cfbe787ed7"
server_key_version: "encryptionkey@0"
I20260924 03:02:58.536063 7152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:58.543370 7152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:58.545269 7152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:58.548405 7152 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40835
I20260924 03:02:58.548799 7152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/master-2/data/info.pb
I20260924 03:02:58.551719 7325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40835 every 8 connection(s)
I20260924 03:02:58.552559 7326 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:02:58.553206 7326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60: Bootstrap starting.
I20260924 03:02:58.555809 7326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:02:58.556031 7326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60: Bootstrap complete.
I20260924 03:02:58.556258 7326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: true } }
I20260924 03:02:58.556366 7326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6e6eae6748bd4849aa05bad1e22daf60, State: Initialized, Role: LEARNER
I20260924 03:02:58.556417 7326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: true } }
I20260924 03:02:58.556633 7326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:02:58.556627 7329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: true } } }
I20260924 03:02:58.556856 7329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [sys.catalog]: This master's current role is: LEARNER
I20260924 03:02:58.558149 7340 catalog_manager.cc:1259] Loaded cluster ID: 66173977c15d45668f6c482930c04bf9
I20260924 03:02:58.558256 7340 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60: loading cluster ID for follower catalog manager: success
I20260924 03:02:58.559998 7340 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60: acquiring CA information for follower catalog manager: success
I20260924 03:02:58.560322 7340 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 03:02:58.569054 7245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:58.569260 7245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:58.569283 7245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:58.570469 7245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:58.570518 7245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:58.570547 7245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:02:58.572139 7245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35349
--tserver_master_addrs=127.0.98.62:36379,127.0.98.61:45601,127.0.98.60:40835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:58.572394 7245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:58.572697 7245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:58.577924 7346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:58.577986 7348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:58.577989 7345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:58.578589 7245 server_base.cc:1034] running on GCE node
I20260924 03:02:58.578819 7245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:58.579092 7245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:58.580258 7245 hybrid_clock.cc:648] HybridClock initialized: now 1790218978580244 us; error 28 us; skew 500 ppm
Sep 24 03:02:58 dist-test-slave-wgsb krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218978, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:58.583137 7245 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 03:02:58.583578 7245 webserver.cc:460] Webserver started at http://127.0.98.1:35349/ using document root <none> and password file <none>
I20260924 03:02:58.583868 7245 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:58.583963 7245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:58.585381 7245 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:02:58.586072 7355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:58.586294 7245 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:02:58.586385 7245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "7bce5a68994249e7884543c4c7276d3f"
format_stamp: "Formatted at 2026-09-24 03:02:57 on dist-test-slave-wgsb"
server_key: "cb7bcf987ba551ee2b15ff670afc2213"
server_key_iv: "5b5cc3982bf652b153474dc824457491"
server_key_version: "encryptionkey@0"
I20260924 03:02:58.586751 7245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:58.596768 7245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:58.599049 7245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:58.599211 7245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:58.599447 7245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:58.599798 7245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:58.599853 7245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:58.599897 7245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:58.599932 7245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:58.607426 7245 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33289
I20260924 03:02:58.607465 7468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33289 every 8 connection(s)
I20260924 03:02:58.607873 7245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:02:58.613261 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 7245
I20260924 03:02:58.613440 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 6696
Sep 24 03:02:58 dist-test-slave-wgsb krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218978, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 03:02:58 dist-test-slave-wgsb krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218978, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 03:02:58 dist-test-slave-wgsb krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218978, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 03:02:58.620638 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.608640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48419 (local address 127.0.98.61:45601)
0924 03:02:58.608696 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.608699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.608906 (+ 207us) server_negotiation.cc:407] Connection header received
0924 03:02:58.609993 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.609994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.610006 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.610059 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.611549 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.612280 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.613032 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.613355 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.613427 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.616084 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.616085 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.616086 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.616094 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.617635 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.619366 (+ 1731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.619367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.619367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.619433 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.619806 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.619807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.619807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.619962 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.620010 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.620413 (+ 403us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.620504 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 03:02:58.620676 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.609084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50613 (local address 127.0.98.62:36379)
0924 03:02:58.609139 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.609140 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.609148 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:02:58.610009 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.610010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.610022 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.610053 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.610900 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.611408 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.612862 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.613172 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.613209 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.617308 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.617309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.617310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.617318 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.618991 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.619685 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.619687 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.619687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.619747 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.619960 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.619961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.619961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.620118 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.620168 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.620438 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.620528 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 03:02:58.621280 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.608623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45601 (local address 127.0.98.1:48419)
0924 03:02:58.608793 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:58.608826 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:58.609885 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:58.610127 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:58.610133 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:58.610590 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:58.610802 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.610807 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.612367 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.612368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.612924 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.612927 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.613481 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.613482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.613554 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.614046 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:58.614056 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:58.615985 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:58.617750 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.617757 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.617760 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.619307 (+ 1547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.619501 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.619502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.619503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.619672 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.620062 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:58.620069 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:58.620245 (+ 176us) client_negotiation.cc:769] Sending connection context
0924 03:02:58.620771 (+ 526us) client_negotiation.cc:240] Negotiation successful
0924 03:02:58.621054 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":87,"thread_start_us":86,"threads_started":1}
I20260924 03:02:58.621287 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.608906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:50613)
0924 03:02:58.609020 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:58.609030 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:58.609908 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:58.610145 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:58.610146 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:58.610601 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:58.610800 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.610805 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.611540 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.611541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.612767 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.612769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.613289 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.613290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.613425 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.614168 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:58.614173 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:58.617210 (+ 3037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:58.619074 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.619075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.619075 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.619563 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.619830 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.619830 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.619831 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.619892 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.620213 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:58.620214 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:58.620269 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 03:02:58.620767 (+ 498us) client_negotiation.cc:240] Negotiation successful
0924 03:02:58.620988 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":73,"threads_started":1}
I20260924 03:02:58.621866 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.608624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40835 (local address 127.0.98.1:58797)
0924 03:02:58.608782 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:58.608824 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:58.609905 (+ 1081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:58.610148 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:58.610149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:58.610591 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:58.610802 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.610807 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.613441 (+ 2634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.613441 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.614294 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.614296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.614787 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.614788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.614847 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.615281 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:58.615286 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:58.618235 (+ 2949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:58.620336 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.620336 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.620337 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.620535 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.620967 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.620968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.620968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.621036 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.621374 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:58.621375 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:58.621425 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 03:02:58.621485 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 03:02:58.621600 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":6,"thread_start_us":100,"threads_started":1}
I20260924 03:02:58.621958 7470 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36379
I20260924 03:02:58.622043 7470 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:58.622148 7471 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45601
I20260924 03:02:58.622190 7471 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:58.622184 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34985
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36379
--webserver_port=33001
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:36379,127.0.98.61:45601
--tserver_master_addrs=127.0.98.62:36379,127.0.98.61:45601,127.0.98.60:40835 with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:02:58.622225 7470 heartbeater.cc:511] Master 127.0.98.62:36379 requested a full tablet report, sending...
I20260924 03:02:58.622256 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.608880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58797 (local address 127.0.98.60:40835)
0924 03:02:58.608997 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.608999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.609007 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:02:58.610051 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.610052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.610064 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.610268 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.612387 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.613343 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.614398 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.614707 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.614788 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.618329 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.618330 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.618330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.618338 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.620248 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.620617 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.620618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.620618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.620673 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.621110 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.621111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.621111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.621288 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.622032 (+ 744us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.622076 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.622163 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":52,"threads_started":1}
I20260924 03:02:58.622260 7471 heartbeater.cc:511] Master 127.0.98.61:45601 requested a full tablet report, sending...
I20260924 03:02:58.623054 7098 ts_manager.cc:194] Registered new tserver with Master: 7bce5a68994249e7884543c4c7276d3f (127.0.98.1:33289)
I20260924 03:02:58.623636 6500 ts_manager.cc:194] Re-registered known tserver with Master: 7bce5a68994249e7884543c4c7276d3f (127.0.98.1:33289)
I20260924 03:02:58.624032 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50613
I20260924 03:02:58.624158 7469 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40835
I20260924 03:02:58.624212 7469 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:58.624270 7469 heartbeater.cc:511] Master 127.0.98.60:40835 requested a full tablet report, sending...
I20260924 03:02:58.624997 7290 ts_manager.cc:194] Registered new tserver with Master: 7bce5a68994249e7884543c4c7276d3f (127.0.98.1:33289)
W20260924 03:02:58.705188 7477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:58.705435 7477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:58.705477 7477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:58.706651 7477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:58.706701 7477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:58.706729 7477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:02:58.708244 7477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33001
--tserver_master_addrs=127.0.98.62:36379,127.0.98.61:45601,127.0.98.60:40835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:58.708436 7477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:58.708622 7477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:58.713882 7482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:58.713811 7483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:58.713922 7477 server_base.cc:1034] running on GCE node
W20260924 03:02:58.713811 7485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:58.714249 7477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:58.714519 7477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:58.715658 7477 hybrid_clock.cc:648] HybridClock initialized: now 1790218978715634 us; error 40 us; skew 500 ppm
Sep 24 03:02:58 dist-test-slave-wgsb krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218978, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:58.718282 7477 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 03:02:58.718753 7477 webserver.cc:460] Webserver started at http://127.0.98.2:33001/ using document root <none> and password file <none>
I20260924 03:02:58.718997 7477 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:58.719077 7477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:58.720674 7477 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:58.721457 7492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:58.721648 7477 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:58.721755 7477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "a7ccdcca74be438cb9cd448857cbc55a"
format_stamp: "Formatted at 2026-09-24 03:02:57 on dist-test-slave-wgsb"
server_key: "b3b2e41d75f2a5ac18738e745eed9f97"
server_key_iv: "5d1c818783f79d83e3523d2ba2f619e1"
server_key_version: "encryptionkey@0"
I20260924 03:02:58.722200 7477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:58.744333 7477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:58.746899 7477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:58.747046 7477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:58.747256 7477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:58.747715 7477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:58.747758 7477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:58.747813 7477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:58.747841 7477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:58.754343 7477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34985
I20260924 03:02:58.754406 7605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34985 every 8 connection(s)
I20260924 03:02:58.754755 7477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:02:58.757762 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 7477
I20260924 03:02:58.757930 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 6830
Sep 24 03:02:58 dist-test-slave-wgsb krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218978, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 03:02:58 dist-test-slave-wgsb krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218978, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 03:02:58.764214 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43013
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36379
--webserver_port=37199
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:36379,127.0.98.61:45601
--tserver_master_addrs=127.0.98.62:36379,127.0.98.61:45601,127.0.98.60:40835 with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:02:58 dist-test-slave-wgsb krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218978, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:02:58.767947 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.755532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52841 (local address 127.0.98.60:40835)
0924 03:02:58.755576 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.755578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.755818 (+ 240us) server_negotiation.cc:407] Connection header received
0924 03:02:58.756998 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.756999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.757013 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.757066 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.757924 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.758779 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.760567 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.760923 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.760963 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.764204 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.764205 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.764206 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.764213 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.766144 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.766640 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.766641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.766642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.766709 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.767025 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.767026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.767026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.767145 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.767195 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.767752 (+ 557us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.767846 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 03:02:58.768654 7610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.755466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40835 (local address 127.0.98.2:52841)
0924 03:02:58.755659 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:58.755701 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:58.756914 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:58.757158 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:58.757158 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:58.757517 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:58.757792 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.757798 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.758895 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.758896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.760452 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.760454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.761064 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.761065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.761220 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.761897 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:58.761909 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:58.764044 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:58.766226 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.766233 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.766236 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.766540 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.766784 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.766785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.766786 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.766929 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.767216 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:58.767222 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:58.767400 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 03:02:58.767509 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 03:02:58.768387 (+ 878us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":121,"thread_start_us":94,"threads_started":1}
I20260924 03:02:58.769297 7606 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40835
I20260924 03:02:58.769397 7606 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:58.769613 7606 heartbeater.cc:511] Master 127.0.98.60:40835 requested a full tablet report, sending...
I20260924 03:02:58.769732 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.755454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56209 (local address 127.0.98.61:45601)
0924 03:02:58.755532 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.755534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.755796 (+ 262us) server_negotiation.cc:407] Connection header received
0924 03:02:58.756913 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.756914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.756926 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.756998 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.757912 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.758712 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.761401 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.761765 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.761806 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.765150 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.765152 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.765152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.765160 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.766962 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.768506 (+ 1544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.768507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.768507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.768561 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.769171 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.769172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.769172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.769312 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.769364 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.769553 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.769633 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 03:02:58.769865 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.755948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33099 (local address 127.0.98.62:36379)
0924 03:02:58.755997 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.755999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.756008 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:02:58.756931 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.756933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.756946 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.757000 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.757941 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.758794 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.760567 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.760913 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.760962 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.766049 (+ 5087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.766050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.766051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.766059 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.768223 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.769045 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.769047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.769047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.769135 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.769327 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.769328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.769328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.769472 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.769517 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.769710 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.769783 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 03:02:58.770033 7611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.755466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45601 (local address 127.0.98.2:56209)
0924 03:02:58.755662 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:58.755701 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:58.756801 (+ 1100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:58.757150 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:58.757154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:58.757459 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:58.757793 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.757799 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.758864 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.758865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.761296 (+ 2431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.761299 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.761881 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.761882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.761988 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.762425 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:58.762430 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:58.765058 (+ 2628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:58.767370 (+ 2312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.767371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.767372 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.768416 (+ 1044us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.768624 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.768625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.768625 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.769091 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.769397 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:58.769398 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:58.769435 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 03:02:58.769805 (+ 370us) client_negotiation.cc:240] Negotiation successful
0924 03:02:58.769901 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":1275,"thread_start_us":96,"threads_started":1}
I20260924 03:02:58.770059 7290 ts_manager.cc:194] Registered new tserver with Master: a7ccdcca74be438cb9cd448857cbc55a (127.0.98.2:34985)
I20260924 03:02:58.770331 7608 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45601
I20260924 03:02:58.770375 7608 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:58.770422 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.755782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:33099)
0924 03:02:58.755886 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:58.755896 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:58.756821 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:58.757156 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:58.757157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:58.757514 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:58.757792 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.757798 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.759008 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.759009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.760452 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.760454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.761062 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.761064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.761220 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.762011 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:58.762016 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:58.765941 (+ 3925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:58.768306 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.768307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.768308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.768976 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.769207 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.769207 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.769208 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.769256 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.769564 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:58.769565 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:58.769599 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 03:02:58.770001 (+ 402us) client_negotiation.cc:240] Negotiation successful
0924 03:02:58.770097 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":209,"thread_start_us":60,"threads_started":1}
I20260924 03:02:58.770445 7608 heartbeater.cc:511] Master 127.0.98.61:45601 requested a full tablet report, sending...
I20260924 03:02:58.770705 7098 ts_manager.cc:194] Registered new tserver with Master: a7ccdcca74be438cb9cd448857cbc55a (127.0.98.2:34985)
I20260924 03:02:58.770794 7607 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36379
I20260924 03:02:58.770830 7607 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:58.770885 7607 heartbeater.cc:511] Master 127.0.98.62:36379 requested a full tablet report, sending...
I20260924 03:02:58.771085 6500 ts_manager.cc:194] Re-registered known tserver with Master: a7ccdcca74be438cb9cd448857cbc55a (127.0.98.2:34985)
I20260924 03:02:58.771443 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33099
W20260924 03:02:58.866914 7613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:02:58.867152 7613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:02:58.867190 7613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:02:58.868412 7613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:02:58.868480 7613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:02:58.868527 7613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:02:58.870007 7613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37199
--tserver_master_addrs=127.0.98.62:36379,127.0.98.61:45601,127.0.98.60:40835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:02:58.870278 7613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:02:58.870512 7613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:02:58.875645 7619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:58.875757 7621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:02:58.875645 7618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:02:58.875953 7613 server_base.cc:1034] running on GCE node
I20260924 03:02:58.876200 7613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:02:58.876467 7613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:02:58.877635 7613 hybrid_clock.cc:648] HybridClock initialized: now 1790218978877623 us; error 32 us; skew 500 ppm
Sep 24 03:02:58 dist-test-slave-wgsb krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218978, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:02:58.880300 7613 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 03:02:58.880788 7613 webserver.cc:460] Webserver started at http://127.0.98.3:37199/ using document root <none> and password file <none>
I20260924 03:02:58.881026 7613 fs_manager.cc:359] Metadata directory not provided
I20260924 03:02:58.881110 7613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:02:58.882565 7613 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:02:58.883247 7628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:58.883464 7613 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 03:02:58.883569 7613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "b5a56c6090d644fb8935dc68bbaeedd8"
format_stamp: "Formatted at 2026-09-24 03:02:57 on dist-test-slave-wgsb"
server_key: "f431be25d1796b692dfb4e9877e279d2"
server_key_iv: "e2e80e3c0073c05552abd2c5eab6e06f"
server_key_version: "encryptionkey@0"
I20260924 03:02:58.883994 7613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:02:58.907636 7613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:02:58.910081 7613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:02:58.910254 7613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:02:58.910509 7613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:02:58.910820 7613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:02:58.910874 7613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:58.910915 7613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:02:58.910986 7613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:02:58.917521 7613 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43013
I20260924 03:02:58.917565 7741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43013 every 8 connection(s)
I20260924 03:02:58.917917 7613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:02:58.923723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 7613
Sep 24 03:02:58 dist-test-slave-wgsb krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218978, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 03:02:58 dist-test-slave-wgsb krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218978, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 03:02:58 dist-test-slave-wgsb krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218978, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 03:02:58.931893 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.918634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57279 (local address 127.0.98.61:45601)
0924 03:02:58.918688 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.918690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.919158 (+ 468us) server_negotiation.cc:407] Connection header received
0924 03:02:58.920354 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.920355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.920368 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.920401 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.921350 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.922136 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.923279 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.923545 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.924138 (+ 593us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.928924 (+ 4786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.928925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.928925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.928932 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.930666 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.931044 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.931045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.931046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.931099 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.931287 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.931288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.931289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.931411 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.931466 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.931720 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.931794 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 03:02:58.931967 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.918638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40835 (local address 127.0.98.3:49875)
0924 03:02:58.918803 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:58.918846 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:58.920229 (+ 1383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:58.920506 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:58.920512 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:58.920991 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:58.921239 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.921245 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.922243 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.922244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.923196 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.923198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.924171 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.924172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.924246 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.924815 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:58.924820 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:58.928233 (+ 3413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:58.930108 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.930116 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.930118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.930471 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.930688 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.930688 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.930689 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.930828 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.931125 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:58.931131 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:58.931346 (+ 215us) client_negotiation.cc:769] Sending connection context
0924 03:02:58.931473 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 03:02:58.931710 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":72,"thread_start_us":96,"threads_started":1}
I20260924 03:02:58.932166 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.918938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:50797)
0924 03:02:58.919105 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:58.919114 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:58.920228 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:58.920505 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:58.920508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:58.920956 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:58.921239 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.921243 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.922077 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.922078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.922913 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.922915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.923372 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.923373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.923519 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.924451 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:58.924461 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:58.927936 (+ 3475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:58.931059 (+ 3123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.931060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.931060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.931268 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.931473 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.931474 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.931474 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.931528 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.931825 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:58.931826 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:58.931865 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 03:02:58.931924 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 03:02:58.932057 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":81,"thread_start_us":75,"threads_started":1}
I20260924 03:02:58.932593 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.918637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45601 (local address 127.0.98.3:57279)
0924 03:02:58.919026 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:58.919090 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:58.920229 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:58.920506 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:58.920512 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:58.920957 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:02:58.921239 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.921245 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.922243 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.922244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.923196 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.923198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.924142 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.924144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.924210 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.924685 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:02:58.924690 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:02:58.928830 (+ 4140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:02:58.930748 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.930749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.930750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.930956 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.931161 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:02:58.931162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:02:58.931162 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:02:58.931227 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:02:58.931493 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:02:58.931494 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:02:58.931537 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 03:02:58.931632 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 03:02:58.932234 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":590,"thread_start_us":99,"threads_started":1}
I20260924 03:02:58.932865 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.918679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49875 (local address 127.0.98.60:40835)
0924 03:02:58.918727 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.918729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.918960 (+ 231us) server_negotiation.cc:407] Connection header received
0924 03:02:58.920354 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.920355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.920368 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.920401 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.921350 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.922152 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.923647 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.923966 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.924039 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.928339 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.928340 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.928341 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.928349 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.930020 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.930560 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.930562 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.930563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.930619 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.930899 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.930900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.930901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.931039 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.931100 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.932702 (+ 1602us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.932779 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:02:58.932945 7744 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45601
I20260924 03:02:58.933058 7744 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:58.933140 7742 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40835
I20260924 03:02:58.933187 7742 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:58.933260 7744 heartbeater.cc:511] Master 127.0.98.61:45601 requested a full tablet report, sending...
I20260924 03:02:58.933265 7742 heartbeater.cc:511] Master 127.0.98.60:40835 requested a full tablet report, sending...
I20260924 03:02:58.933775 7290 ts_manager.cc:194] Registered new tserver with Master: b5a56c6090d644fb8935dc68bbaeedd8 (127.0.98.3:43013)
I20260924 03:02:58.934127 7098 ts_manager.cc:194] Registered new tserver with Master: b5a56c6090d644fb8935dc68bbaeedd8 (127.0.98.3:43013)
I20260924 03:02:58.935976 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.919021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50797 (local address 127.0.98.62:36379)
0924 03:02:58.919074 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.919076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.919179 (+ 103us) server_negotiation.cc:407] Connection header received
0924 03:02:58.920336 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.920337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.920349 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.920414 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.921333 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.921987 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.923017 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.923227 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.923308 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.928021 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.928022 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.928023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.928031 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.930979 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.931349 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.931350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.931350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.931408 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.931593 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.931594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.931594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.931740 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.931800 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.935759 (+ 3959us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.935846 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 03:02:58.936278 7743 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36379
I20260924 03:02:58.936329 7743 heartbeater.cc:464] Registering TS with master...
I20260924 03:02:58.936399 7743 heartbeater.cc:511] Master 127.0.98.62:36379 requested a full tablet report, sending...
I20260924 03:02:58.936625 6500 ts_manager.cc:194] Re-registered known tserver with Master: b5a56c6090d644fb8935dc68bbaeedd8 (127.0.98.3:43013)
I20260924 03:02:58.937142 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50797
I20260924 03:02:58.937551 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.927416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51872 (local address 127.0.98.61:45601)
0924 03:02:58.927551 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.927553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.929299 (+ 1746us) server_negotiation.cc:407] Connection header received
0924 03:02:58.929318 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.929318 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.929329 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.929378 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.930506 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.931294 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.932302 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.932652 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.932719 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.934143 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.934144 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.934144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.934151 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.936121 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.936556 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.936557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.936557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.936614 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.936812 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.936813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.936814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.936992 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.937120 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.937381 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.937465 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260924 03:02:58.938019 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.924900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40174 (local address 127.0.98.60:40835)
0924 03:02:58.925018 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.925020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.925379 (+ 359us) server_negotiation.cc:407] Connection header received
0924 03:02:58.925508 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.925509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.925523 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.925563 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.926334 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.926945 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.928045 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.928364 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.928409 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.934222 (+ 5813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.934224 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.934224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.934230 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.935983 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.936664 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.936665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.936665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.936721 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.936918 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.936919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.936919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.937033 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.937102 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.937724 (+ 622us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.937905 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":69,"threads_started":1}
I20260924 03:02:58.938223 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.925024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54232 (local address 127.0.98.62:36379)
0924 03:02:58.925164 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.925166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.926467 (+ 1301us) server_negotiation.cc:407] Connection header received
0924 03:02:58.926493 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.926493 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.926505 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.926562 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:58.927431 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.928231 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.929240 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.929537 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.929598 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.935004 (+ 5406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:58.935005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:58.935005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:58.935012 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:58.936802 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.937190 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.937191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.937191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.937249 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:58.937435 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:58.937436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:58.937437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:58.937585 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:58.937677 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.937987 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.938065 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260924 03:02:58.939559 6500 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:02:58.941097 6500 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:02:58.950127 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.944813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34985 (local address 127.0.0.1:55068)
0924 03:02:58.944977 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:58.944986 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:58.945111 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:58.945562 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:58.945563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:58.945566 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:58.945623 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.945625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.946933 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.946933 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.948202 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.948206 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.949854 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.949855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.949904 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.949907 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:02:58.949994 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 03:02:58.950036 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":86,"thread_start_us":93,"threads_started":1}
I20260924 03:02:58.950137 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.945164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55068 (local address 127.0.98.2:34985)
0924 03:02:58.945357 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.945361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.945371 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:02:58.945431 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.945433 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.945450 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.945566 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:58.945705 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.946822 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.948302 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.949046 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.949135 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.949202 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.950012 (+ 810us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.950058 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":83,"threads_started":1}
I20260924 03:02:58.950328 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.944813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33289 (local address 127.0.0.1:43556)
0924 03:02:58.944871 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:58.944883 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:58.944920 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:58.945646 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:58.945647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:58.945649 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:58.945698 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.945700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.946919 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.946920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.948202 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.948206 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.950113 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.950113 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.950168 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.950171 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:02:58.950200 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 03:02:58.950246 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":35}
I20260924 03:02:58.950505 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.945030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43013 (local address 127.0.0.1:50864)
0924 03:02:58.945340 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:58.945352 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:58.945422 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:58.945646 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:58.945647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:58.945649 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:58.945705 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.945706 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.947995 (+ 2289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.947996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.949033 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:58.949036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.949801 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.949802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:58.949863 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.949868 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:02:58.949924 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 03:02:58.949974 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":6,"thread_start_us":66,"threads_started":1}
I20260924 03:02:58.950706 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.945015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43556 (local address 127.0.98.1:33289)
0924 03:02:58.945446 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.945450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.945482 (+ 32us) server_negotiation.cc:407] Connection header received
0924 03:02:58.945520 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.945522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.945539 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.945647 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:58.945757 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.946835 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.949237 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.950021 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.950426 (+ 405us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.950527 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.950555 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.950602 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":96,"threads_started":1}
I20260924 03:02:58.950897 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:58.945205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50864 (local address 127.0.98.3:43013)
0924 03:02:58.945370 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:58.945374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:58.945430 (+ 56us) server_negotiation.cc:407] Connection header received
0924 03:02:58.945499 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:58.945501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:58.945515 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:58.945584 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:58.946952 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.947905 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.949122 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:58.949706 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:58.949775 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:58.949822 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:58.950087 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 03:02:58.950132 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":70,"threads_started":1}
I20260924 03:02:58.950949 7540 tablet_service.cc:1467] Processing CreateTablet for tablet 667a869c188f4bc68538e9691019f999 (DEFAULT_TABLE table=test-table [id=002f401c4389415caaefa534c2d08d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:58.951318 7540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 667a869c188f4bc68538e9691019f999. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:58.951308 7403 tablet_service.cc:1467] Processing CreateTablet for tablet 667a869c188f4bc68538e9691019f999 (DEFAULT_TABLE table=test-table [id=002f401c4389415caaefa534c2d08d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:58.951431 7676 tablet_service.cc:1467] Processing CreateTablet for tablet 667a869c188f4bc68538e9691019f999 (DEFAULT_TABLE table=test-table [id=002f401c4389415caaefa534c2d08d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:02:58.951613 7403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 667a869c188f4bc68538e9691019f999. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:58.951757 7676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 667a869c188f4bc68538e9691019f999. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:02:58.954520 7768 tablet_bootstrap.cc:492] T 667a869c188f4bc68538e9691019f999 P b5a56c6090d644fb8935dc68bbaeedd8: Bootstrap starting.
I20260924 03:02:58.955237 7768 tablet_bootstrap.cc:654] T 667a869c188f4bc68538e9691019f999 P b5a56c6090d644fb8935dc68bbaeedd8: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:58.955243 7770 tablet_bootstrap.cc:492] T 667a869c188f4bc68538e9691019f999 P a7ccdcca74be438cb9cd448857cbc55a: Bootstrap starting.
I20260924 03:02:58.955711 7768 log.cc:824] T 667a869c188f4bc68538e9691019f999 P b5a56c6090d644fb8935dc68bbaeedd8: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:58.955798 7769 tablet_bootstrap.cc:492] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f: Bootstrap starting.
I20260924 03:02:58.955936 7770 tablet_bootstrap.cc:654] T 667a869c188f4bc68538e9691019f999 P a7ccdcca74be438cb9cd448857cbc55a: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:58.956432 7769 tablet_bootstrap.cc:654] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f: Neither blocks nor log segments found. Creating new log.
I20260924 03:02:58.956667 7770 log.cc:824] T 667a869c188f4bc68538e9691019f999 P a7ccdcca74be438cb9cd448857cbc55a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:58.956714 7769 log.cc:824] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f: Log is configured to *not* fsync() on all Append() calls
I20260924 03:02:58.956980 7768 tablet_bootstrap.cc:492] T 667a869c188f4bc68538e9691019f999 P b5a56c6090d644fb8935dc68bbaeedd8: No bootstrap required, opened a new log
I20260924 03:02:58.957101 7768 ts_tablet_manager.cc:1397] T 667a869c188f4bc68538e9691019f999 P b5a56c6090d644fb8935dc68bbaeedd8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 03:02:58.957463 7770 tablet_bootstrap.cc:492] T 667a869c188f4bc68538e9691019f999 P a7ccdcca74be438cb9cd448857cbc55a: No bootstrap required, opened a new log
I20260924 03:02:58.957547 7770 ts_tablet_manager.cc:1397] T 667a869c188f4bc68538e9691019f999 P a7ccdcca74be438cb9cd448857cbc55a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 03:02:58.957553 7769 tablet_bootstrap.cc:492] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f: No bootstrap required, opened a new log
I20260924 03:02:58.957612 7769 ts_tablet_manager.cc:1397] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:02:58.959084 7770 raft_consensus.cc:348] T 667a869c188f4bc68538e9691019f999 P a7ccdcca74be438cb9cd448857cbc55a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ccdcca74be438cb9cd448857cbc55a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34985 } } peers { permanent_uuid: "7bce5a68994249e7884543c4c7276d3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33289 } } peers { permanent_uuid: "b5a56c6090d644fb8935dc68bbaeedd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43013 } }
I20260924 03:02:58.959214 7770 raft_consensus.cc:374] T 667a869c188f4bc68538e9691019f999 P a7ccdcca74be438cb9cd448857cbc55a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:58.959236 7770 raft_consensus.cc:729] T 667a869c188f4bc68538e9691019f999 P a7ccdcca74be438cb9cd448857cbc55a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7ccdcca74be438cb9cd448857cbc55a, State: Initialized, Role: FOLLOWER
I20260924 03:02:58.959327 7770 consensus_queue.cc:260] T 667a869c188f4bc68538e9691019f999 P a7ccdcca74be438cb9cd448857cbc55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ccdcca74be438cb9cd448857cbc55a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34985 } } peers { permanent_uuid: "7bce5a68994249e7884543c4c7276d3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33289 } } peers { permanent_uuid: "b5a56c6090d644fb8935dc68bbaeedd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43013 } }
I20260924 03:02:58.959568 7770 ts_tablet_manager.cc:1428] T 667a869c188f4bc68538e9691019f999 P a7ccdcca74be438cb9cd448857cbc55a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 03:02:58.959662 7607 heartbeater.cc:503] Master 127.0.98.62:36379 was elected leader, sending a full tablet report...
I20260924 03:02:58.959975 7768 raft_consensus.cc:348] T 667a869c188f4bc68538e9691019f999 P b5a56c6090d644fb8935dc68bbaeedd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ccdcca74be438cb9cd448857cbc55a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34985 } } peers { permanent_uuid: "7bce5a68994249e7884543c4c7276d3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33289 } } peers { permanent_uuid: "b5a56c6090d644fb8935dc68bbaeedd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43013 } }
I20260924 03:02:58.960127 7768 raft_consensus.cc:374] T 667a869c188f4bc68538e9691019f999 P b5a56c6090d644fb8935dc68bbaeedd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:58.960160 7768 raft_consensus.cc:729] T 667a869c188f4bc68538e9691019f999 P b5a56c6090d644fb8935dc68bbaeedd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a56c6090d644fb8935dc68bbaeedd8, State: Initialized, Role: FOLLOWER
I20260924 03:02:58.960115 7769 raft_consensus.cc:348] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ccdcca74be438cb9cd448857cbc55a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34985 } } peers { permanent_uuid: "7bce5a68994249e7884543c4c7276d3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33289 } } peers { permanent_uuid: "b5a56c6090d644fb8935dc68bbaeedd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43013 } }
I20260924 03:02:58.960249 7769 raft_consensus.cc:374] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:02:58.960280 7769 raft_consensus.cc:729] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bce5a68994249e7884543c4c7276d3f, State: Initialized, Role: FOLLOWER
I20260924 03:02:58.960309 7768 consensus_queue.cc:260] T 667a869c188f4bc68538e9691019f999 P b5a56c6090d644fb8935dc68bbaeedd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ccdcca74be438cb9cd448857cbc55a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34985 } } peers { permanent_uuid: "7bce5a68994249e7884543c4c7276d3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33289 } } peers { permanent_uuid: "b5a56c6090d644fb8935dc68bbaeedd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43013 } }
I20260924 03:02:58.960552 7768 ts_tablet_manager.cc:1428] T 667a869c188f4bc68538e9691019f999 P b5a56c6090d644fb8935dc68bbaeedd8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:02:58.960731 7769 consensus_queue.cc:260] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ccdcca74be438cb9cd448857cbc55a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34985 } } peers { permanent_uuid: "7bce5a68994249e7884543c4c7276d3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33289 } } peers { permanent_uuid: "b5a56c6090d644fb8935dc68bbaeedd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43013 } }
I20260924 03:02:58.961053 7769 ts_tablet_manager.cc:1428] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:02:58.961179 7743 heartbeater.cc:503] Master 127.0.98.62:36379 was elected leader, sending a full tablet report...
I20260924 03:02:58.961190 7470 heartbeater.cc:503] Master 127.0.98.62:36379 was elected leader, sending a full tablet report...
W20260924 03:02:59.005571 7609 tablet.cc:2369] T 667a869c188f4bc68538e9691019f999 P a7ccdcca74be438cb9cd448857cbc55a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:02:59.063277 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:59.060533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40188 (local address 127.0.98.60:40835)
0924 03:02:59.060594 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:59.060597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:59.060701 (+ 104us) server_negotiation.cc:407] Connection header received
0924 03:02:59.060758 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:59.060759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:59.060779 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:59.060876 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:59.061024 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:59.061512 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:59.062566 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:59.063004 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:59.063066 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:59.063130 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:59.063191 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 03:02:59.063219 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 03:02:59.063292 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:59.060453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40835 (local address 127.0.0.1:40188)
0924 03:02:59.060626 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:59.060639 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:59.060699 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:59.060895 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:59.060896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:59.060898 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:59.060953 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:59.060955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:59.061627 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:59.061628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:59.062494 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:59.062497 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:59.063081 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:59.063082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:59.063132 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:59.063135 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:02:59.063184 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 03:02:59.063226 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260924 03:02:59.063691 7300 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260924 03:02:59.064256 7777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 105f4753f35f455c9ebfdf86ba7c065c. Latest consensus state: current_term: 1 leader_uuid: "105f4753f35f455c9ebfdf86ba7c065c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: true } } }
I20260924 03:02:59.064422 7777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [sys.catalog]: This master's current role is: LEARNER
W20260924 03:02:59.108855 7472 tablet.cc:2369] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:02:59.168764 7745 tablet.cc:2369] T 667a869c188f4bc68538e9691019f999 P b5a56c6090d644fb8935dc68bbaeedd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:02:59.207306 7776 raft_consensus.cc:482] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:02:59.207562 7776 raft_consensus.cc:504] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ccdcca74be438cb9cd448857cbc55a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34985 } } peers { permanent_uuid: "7bce5a68994249e7884543c4c7276d3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33289 } } peers { permanent_uuid: "b5a56c6090d644fb8935dc68bbaeedd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43013 } }
I20260924 03:02:59.207989 7776 leader_election.cc:288] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7ccdcca74be438cb9cd448857cbc55a (127.0.98.2:34985), b5a56c6090d644fb8935dc68bbaeedd8 (127.0.98.3:43013)
I20260924 03:02:59.211216 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:59.208481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57041 (local address 127.0.98.3:43013)
0924 03:02:59.208553 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:59.208555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:59.208565 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:02:59.208614 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:59.208615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:59.208628 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:59.208681 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:59.208866 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:59.209344 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:59.210336 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:59.210838 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:59.210896 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:59.210905 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:59.211124 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 03:02:59.211166 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 03:02:59.211251 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:59.208382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43013 (local address 127.0.98.1:57041)
0924 03:02:59.208485 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:59.208495 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:59.208569 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:59.208694 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:59.208694 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:59.208696 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:59.208752 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:59.208753 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:59.209427 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:59.209428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:59.210229 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:59.210231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:59.210929 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:59.210929 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:59.211014 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:59.211021 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 03:02:59.211092 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 03:02:59.211141 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":24,"thread_start_us":41,"threads_started":1}
I20260924 03:02:59.211869 7696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667a869c188f4bc68538e9691019f999" candidate_uuid: "7bce5a68994249e7884543c4c7276d3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a56c6090d644fb8935dc68bbaeedd8" is_pre_election: true
I20260924 03:02:59.211958 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:59.208198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34985 (local address 127.0.98.1:51793)
0924 03:02:59.208328 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 03:02:59.208340 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:02:59.208416 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:02:59.208673 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:02:59.208674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:02:59.208676 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:02:59.208750 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:59.208752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:59.209506 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:59.209507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:59.210536 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:02:59.210538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:02:59.211766 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:02:59.211767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:02:59.211815 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:59.211819 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:02:59.211861 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 03:02:59.211900 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":1,"thread_start_us":74,"threads_started":1}
I20260924 03:02:59.211992 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:59.208287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51793 (local address 127.0.98.2:34985)
0924 03:02:59.208353 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:59.208356 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:59.208399 (+ 43us) server_negotiation.cc:407] Connection header received
0924 03:02:59.208544 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:59.208546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:59.208560 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:59.208594 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:02:59.208903 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:59.209363 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:59.210629 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:59.211669 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:59.211749 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:59.211758 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:59.211885 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 03:02:59.211923 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 03:02:59.212025 7696 raft_consensus.cc:2454] T 667a869c188f4bc68538e9691019f999 P b5a56c6090d644fb8935dc68bbaeedd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bce5a68994249e7884543c4c7276d3f in term 0.
I20260924 03:02:59.212270 7357 leader_election.cc:302] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bce5a68994249e7884543c4c7276d3f, b5a56c6090d644fb8935dc68bbaeedd8; no voters:
I20260924 03:02:59.212275 7560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667a869c188f4bc68538e9691019f999" candidate_uuid: "7bce5a68994249e7884543c4c7276d3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7ccdcca74be438cb9cd448857cbc55a" is_pre_election: true
I20260924 03:02:59.212394 7560 raft_consensus.cc:2454] T 667a869c188f4bc68538e9691019f999 P a7ccdcca74be438cb9cd448857cbc55a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bce5a68994249e7884543c4c7276d3f in term 0.
I20260924 03:02:59.212419 7776 raft_consensus.cc:2781] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:02:59.212508 7776 raft_consensus.cc:482] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:02:59.212535 7776 raft_consensus.cc:3037] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:59.213231 7776 raft_consensus.cc:504] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ccdcca74be438cb9cd448857cbc55a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34985 } } peers { permanent_uuid: "7bce5a68994249e7884543c4c7276d3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33289 } } peers { permanent_uuid: "b5a56c6090d644fb8935dc68bbaeedd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43013 } }
I20260924 03:02:59.213372 7776 leader_election.cc:288] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f [CANDIDATE]: Term 1 election: Requested vote from peers a7ccdcca74be438cb9cd448857cbc55a (127.0.98.2:34985), b5a56c6090d644fb8935dc68bbaeedd8 (127.0.98.3:43013)
I20260924 03:02:59.213542 7560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667a869c188f4bc68538e9691019f999" candidate_uuid: "7bce5a68994249e7884543c4c7276d3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7ccdcca74be438cb9cd448857cbc55a"
I20260924 03:02:59.213598 7696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "667a869c188f4bc68538e9691019f999" candidate_uuid: "7bce5a68994249e7884543c4c7276d3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a56c6090d644fb8935dc68bbaeedd8"
I20260924 03:02:59.213640 7560 raft_consensus.cc:3037] T 667a869c188f4bc68538e9691019f999 P a7ccdcca74be438cb9cd448857cbc55a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:59.213693 7696 raft_consensus.cc:3037] T 667a869c188f4bc68538e9691019f999 P b5a56c6090d644fb8935dc68bbaeedd8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:02:59.214357 7560 raft_consensus.cc:2454] T 667a869c188f4bc68538e9691019f999 P a7ccdcca74be438cb9cd448857cbc55a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bce5a68994249e7884543c4c7276d3f in term 1.
I20260924 03:02:59.214428 7696 raft_consensus.cc:2454] T 667a869c188f4bc68538e9691019f999 P b5a56c6090d644fb8935dc68bbaeedd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bce5a68994249e7884543c4c7276d3f in term 1.
I20260924 03:02:59.214560 7357 leader_election.cc:302] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bce5a68994249e7884543c4c7276d3f, a7ccdcca74be438cb9cd448857cbc55a; no voters:
I20260924 03:02:59.214651 7776 raft_consensus.cc:2781] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:02:59.214781 7776 raft_consensus.cc:686] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f [term 1 LEADER]: Becoming Leader. State: Replica: 7bce5a68994249e7884543c4c7276d3f, State: Running, Role: LEADER
I20260924 03:02:59.214890 7776 consensus_queue.cc:237] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ccdcca74be438cb9cd448857cbc55a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34985 } } peers { permanent_uuid: "7bce5a68994249e7884543c4c7276d3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33289 } } peers { permanent_uuid: "b5a56c6090d644fb8935dc68bbaeedd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43013 } }
I20260924 03:02:59.215689 6499 catalog_manager.cc:5495] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bce5a68994249e7884543c4c7276d3f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7bce5a68994249e7884543c4c7276d3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ccdcca74be438cb9cd448857cbc55a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bce5a68994249e7884543c4c7276d3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5a56c6090d644fb8935dc68bbaeedd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43013 } health_report { overall_health: UNKNOWN } } }
I20260924 03:02:59.216326 7059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "105f4753f35f455c9ebfdf86ba7c065c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: true } } }
I20260924 03:02:59.216424 7059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:59.217026 7785 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c: attempting to promote NON_VOTER 6e6eae6748bd4849aa05bad1e22daf60 to VOTER
I20260924 03:02:59.217131 7785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: false } }
I20260924 03:02:59.217491 7108 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 1 FOLLOWER]: Refusing update from remote peer 105f4753f35f455c9ebfdf86ba7c065c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260924 03:02:59.217672 7060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [LEADER]: Connected to new peer: Peer: permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260924 03:02:59.217968 7300 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 1 LEARNER]: Refusing update from remote peer 105f4753f35f455c9ebfdf86ba7c065c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260924 03:02:59.218174 7060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260924 03:02:59.218370 7059 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 6e6eae6748bd4849aa05bad1e22daf60 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: false } } }
I20260924 03:02:59.219072 7059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "105f4753f35f455c9ebfdf86ba7c065c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: false } } }
I20260924 03:02:59.219099 7060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "105f4753f35f455c9ebfdf86ba7c065c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: false } } }
I20260924 03:02:59.219164 7059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:59.219185 7060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:59.219442 7108 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 6e6eae6748bd4849aa05bad1e22daf60 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: false } } }
I20260924 03:02:59.219527 7060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "105f4753f35f455c9ebfdf86ba7c065c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: false } } }
I20260924 03:02:59.219642 7060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c [sys.catalog]: This master's current role is: LEADER
I20260924 03:02:59.219691 7300 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 6e6eae6748bd4849aa05bad1e22daf60 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: false } } }
I20260924 03:02:59.220345 7787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "105f4753f35f455c9ebfdf86ba7c065c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: false } } }
I20260924 03:02:59.220435 7787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:02:59.220502 7777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "105f4753f35f455c9ebfdf86ba7c065c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: false } } }
I20260924 03:02:59.220590 7777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:02:59.233716 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:59.231130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43568 (local address 127.0.98.1:33289)
0924 03:02:59.231222 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:59.231224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:59.231233 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:02:59.231264 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:59.231264 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:59.231281 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:59.231351 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:02:59.231501 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:59.232046 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:59.232890 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:59.233165 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:59.233224 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:59.233359 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:02:59.233457 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:02:59.233514 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:59.233598 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 03:02:59.233615 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 03:02:59.235646 7696 raft_consensus.cc:1261] T 667a869c188f4bc68538e9691019f999 P b5a56c6090d644fb8935dc68bbaeedd8 [term 1 FOLLOWER]: Refusing update from remote peer 7bce5a68994249e7884543c4c7276d3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:02:59.235646 7560 raft_consensus.cc:1261] T 667a869c188f4bc68538e9691019f999 P a7ccdcca74be438cb9cd448857cbc55a [term 1 FOLLOWER]: Refusing update from remote peer 7bce5a68994249e7884543c4c7276d3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:02:59.236047 7776 consensus_queue.cc:1035] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a56c6090d644fb8935dc68bbaeedd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:02:59.236198 7783 consensus_queue.cc:1035] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7ccdcca74be438cb9cd448857cbc55a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:02:59.239023 7789 mvcc.cc:205] Tried to move back new op lower bound from 7332736938946080768 to 7332736938864652288. Current Snapshot: MvccSnapshot[applied={T|T < 7332736938946080768}]
I20260924 03:02:59.240316 7790 mvcc.cc:205] Tried to move back new op lower bound from 7332736938946080768 to 7332736938864652288. Current Snapshot: MvccSnapshot[applied={T|T < 7332736938946080768}]
I20260924 03:02:59.242990 6499 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:02:59.243080 6499 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:02:59.244566 6499 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 105f4753f35f455c9ebfdf86ba7c065c: Sending DeleteTablet for 3 replicas of tablet 667a869c188f4bc68538e9691019f999
I20260924 03:02:59.244851 7403 tablet_service.cc:1514] Processing DeleteTablet for tablet 667a869c188f4bc68538e9691019f999 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:02:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43556
I20260924 03:02:59.244938 7676 tablet_service.cc:1514] Processing DeleteTablet for tablet 667a869c188f4bc68538e9691019f999 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:02:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50864
I20260924 03:02:59.245095 7540 tablet_service.cc:1514] Processing DeleteTablet for tablet 667a869c188f4bc68538e9691019f999 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:02:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55068
I20260924 03:02:59.245306 7798 tablet_replica.cc:330] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f: stopping tablet replica
I20260924 03:02:59.245306 7800 tablet_replica.cc:330] T 667a869c188f4bc68538e9691019f999 P a7ccdcca74be438cb9cd448857cbc55a: stopping tablet replica
I20260924 03:02:59.245404 7798 raft_consensus.cc:2229] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:02:59.245513 7800 raft_consensus.cc:2229] T 667a869c188f4bc68538e9691019f999 P a7ccdcca74be438cb9cd448857cbc55a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:02:59.245553 7798 raft_consensus.cc:2258] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:02:59.245693 7800 raft_consensus.cc:2258] T 667a869c188f4bc68538e9691019f999 P a7ccdcca74be438cb9cd448857cbc55a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:02:59.245769 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 6468
I20260924 03:02:59.246176 7798 ts_tablet_manager.cc:1905] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:02:59.246743 7800 ts_tablet_manager.cc:1905] T 667a869c188f4bc68538e9691019f999 P a7ccdcca74be438cb9cd448857cbc55a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:02:59.247927 7798 ts_tablet_manager.cc:1918] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:02:59.248016 7798 log.cc:1196] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-0/wal/wals/667a869c188f4bc68538e9691019f999
I20260924 03:02:59.248278 7798 ts_tablet_manager.cc:1939] T 667a869c188f4bc68538e9691019f999 P 7bce5a68994249e7884543c4c7276d3f: Deleting consensus metadata
I20260924 03:02:59.248647 7800 ts_tablet_manager.cc:1918] T 667a869c188f4bc68538e9691019f999 P a7ccdcca74be438cb9cd448857cbc55a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:02:59.248787 7800 log.cc:1196] T 667a869c188f4bc68538e9691019f999 P a7ccdcca74be438cb9cd448857cbc55a: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/wal/wals/667a869c188f4bc68538e9691019f999
I20260924 03:02:59.249042 7800 ts_tablet_manager.cc:1939] T 667a869c188f4bc68538e9691019f999 P a7ccdcca74be438cb9cd448857cbc55a: Deleting consensus metadata
W20260924 03:02:59.252405 7493 connection.cc:535] server connection from 127.0.0.1:55068 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 03:02:59.252405 7358 connection.cc:535] server connection from 127.0.0.1:43556 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 03:02:59.256860 7799 tablet_replica.cc:330] T 667a869c188f4bc68538e9691019f999 P b5a56c6090d644fb8935dc68bbaeedd8: stopping tablet replica
I20260924 03:02:59.256999 7799 raft_consensus.cc:2229] T 667a869c188f4bc68538e9691019f999 P b5a56c6090d644fb8935dc68bbaeedd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:02:59.257166 7799 raft_consensus.cc:2258] T 667a869c188f4bc68538e9691019f999 P b5a56c6090d644fb8935dc68bbaeedd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:02:59.257692 7799 ts_tablet_manager.cc:1905] T 667a869c188f4bc68538e9691019f999 P b5a56c6090d644fb8935dc68bbaeedd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:02:59.260305 7799 ts_tablet_manager.cc:1918] T 667a869c188f4bc68538e9691019f999 P b5a56c6090d644fb8935dc68bbaeedd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:02:59.260416 7799 log.cc:1196] T 667a869c188f4bc68538e9691019f999 P b5a56c6090d644fb8935dc68bbaeedd8: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/wal/wals/667a869c188f4bc68538e9691019f999
I20260924 03:02:59.260752 7799 ts_tablet_manager.cc:1939] T 667a869c188f4bc68538e9691019f999 P b5a56c6090d644fb8935dc68bbaeedd8: Deleting consensus metadata
W20260924 03:02:59.261181 7629 connection.cc:441] server connection from 127.0.0.1:50864 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50864 (request call id 1) could send its response
I20260924 03:02:59.263281 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:59.254079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51888 (local address 127.0.98.61:45601)
0924 03:02:59.254117 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:59.254119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:59.254347 (+ 228us) server_negotiation.cc:407] Connection header received
0924 03:02:59.254443 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:59.254444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:59.254458 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:59.254489 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:59.255231 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:59.256053 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:59.257175 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:59.257498 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:59.257545 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:59.259102 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:59.259104 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:59.259104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:59.259112 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:59.261281 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:59.262413 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:59.262415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:59.262415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:59.262469 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:59.262664 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:59.262665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:59.262665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:59.262799 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:59.262873 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:59.263089 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 03:02:59.263176 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:02:59.264040 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:59.253933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40206 (local address 127.0.98.60:40835)
0924 03:02:59.253978 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:59.253980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:59.254268 (+ 288us) server_negotiation.cc:407] Connection header received
0924 03:02:59.254380 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:59.254380 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:59.254391 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:59.254425 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:59.255478 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:59.256295 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:59.257620 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:59.257971 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:59.258016 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:59.259774 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:59.259775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:59.259776 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:59.259781 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:59.262358 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:59.262782 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:59.262784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:59.262784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:59.262841 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:59.263090 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:59.263091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:59.263091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:59.263235 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:59.263322 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:59.263851 (+ 529us) server_negotiation.cc:299] Negotiation successful
0924 03:02:59.263938 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 03:02:59.264174 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:02:59.245289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40198 (local address 127.0.98.60:40835)
0924 03:02:59.245332 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 03:02:59.245335 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:02:59.252714 (+ 7379us) server_negotiation.cc:407] Connection header received
0924 03:02:59.252835 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:02:59.252836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:02:59.252846 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:02:59.252934 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:02:59.254665 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:59.255541 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:59.256823 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:02:59.257159 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:02:59.257202 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:02:59.259768 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:02:59.259769 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:02:59.259769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:02:59.259776 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:02:59.262851 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:59.263275 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:59.263276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:59.263277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:59.263327 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:02:59.263546 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:02:59.263546 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:02:59.263547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:02:59.263696 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:02:59.263796 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 03:02:59.263981 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 03:02:59.264066 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 03:03:00.240895 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:00.240576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:47887)
0924 03:03:00.240691 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:00.240757 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":46,"threads_started":1}
I20260924 03:03:00.240902 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:00.240472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:57089)
0924 03:03:00.240607 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:00.240695 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
W20260924 03:03:00.241166 7607 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36379 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
I20260924 03:03:00.241318 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:00.241004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:53915)
0924 03:03:00.241199 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:00.241275 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":51,"threads_started":1}
W20260924 03:03:00.241521 7470 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36379 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
W20260924 03:03:00.241775 7743 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36379 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
I20260924 03:03:00.745257 7812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 105f4753f35f455c9ebfdf86ba7c065c)
I20260924 03:03:00.745337 7813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 105f4753f35f455c9ebfdf86ba7c065c)
I20260924 03:03:00.745417 7813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: false } }
I20260924 03:03:00.745489 7812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: false } }
I20260924 03:03:00.745756 7813 leader_election.cc:288] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 105f4753f35f455c9ebfdf86ba7c065c (127.0.98.62:36379), 6e6eae6748bd4849aa05bad1e22daf60 (127.0.98.60:40835)
I20260924 03:03:00.745918 7812 leader_election.cc:288] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 105f4753f35f455c9ebfdf86ba7c065c (127.0.98.62:36379), 065ca39885b54be2bd9fe99c4d2e3fb6 (127.0.98.61:45601)
I20260924 03:03:00.746165 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:00.745887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.0.1:54696)
0924 03:03:00.746044 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:00.746114 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":48,"threads_started":1}
I20260924 03:03:00.746384 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:00.746123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.0.1:54710)
0924 03:03:00.746240 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:00.746317 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
W20260924 03:03:00.746438 7084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
W20260924 03:03:00.746842 7276 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
I20260924 03:03:00.747164 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:00.747067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.0.1:54722)
0924 03:03:00.747108 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:00.747133 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 03:03:00.747308 7084 leader_election.cc:334] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 105f4753f35f455c9ebfdf86ba7c065c (127.0.98.62:36379): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
I20260924 03:03:00.747443 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:00.747272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.0.1:54728)
0924 03:03:00.747378 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:00.747408 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":61,"threads_started":1}
W20260924 03:03:00.747509 7276 leader_election.cc:334] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 105f4753f35f455c9ebfdf86ba7c065c (127.0.98.62:36379): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
I20260924 03:03:00.749559 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:00.746317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51900 (local address 127.0.98.61:45601)
0924 03:03:00.746419 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:00.746421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:00.746894 (+ 473us) server_negotiation.cc:407] Connection header received
0924 03:03:00.746993 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:00.746994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:00.747005 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:00.747074 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:03:00.747219 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:00.747886 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:00.748813 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:00.749200 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:00.749295 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:00.749307 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:00.749457 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 03:03:00.749501 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":47,"threads_started":1}
I20260924 03:03:00.749553 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:00.746681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45601 (local address 127.0.0.1:51900)
0924 03:03:00.746754 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:00.746767 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:00.746833 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:00.747078 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:00.747079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:00.747082 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:03:00.747133 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:00.747135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:00.747965 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:00.747966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:00.748723 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:00.748725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:00.749297 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:00.749298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:00.749363 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:00.749367 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:03:00.749448 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 03:03:00.749489 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 03:03:00.749696 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:00.746714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40216 (local address 127.0.98.60:40835)
0924 03:03:00.746811 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:00.746813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:00.746822 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:03:00.746850 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:00.746851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:00.746862 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:00.747135 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:03:00.747170 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:00.747899 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:00.748918 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:00.749322 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:00.749422 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:00.749433 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:00.749588 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 03:03:00.749631 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":59,"threads_started":1}
I20260924 03:03:00.749697 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:00.745932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40835 (local address 127.0.0.1:40216)
0924 03:03:00.746134 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:00.746145 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:00.746238 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:00.746981 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:00.746982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:00.746985 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:03:00.747057 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:00.747060 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:00.748003 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:00.748004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:00.748827 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:00.748828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:00.749424 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:00.749426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:00.749499 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:00.749502 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:03:00.749581 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 03:03:00.749625 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":69,"threads_started":1}
I20260924 03:03:00.750165 7108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e6eae6748bd4849aa05bad1e22daf60" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" is_pre_election: true
I20260924 03:03:00.750226 7300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6e6eae6748bd4849aa05bad1e22daf60" is_pre_election: true
I20260924 03:03:00.750319 7300 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 065ca39885b54be2bd9fe99c4d2e3fb6 in term 1.
I20260924 03:03:00.750339 7108 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e6eae6748bd4849aa05bad1e22daf60 in term 1.
I20260924 03:03:00.750480 7085 leader_election.cc:302] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 065ca39885b54be2bd9fe99c4d2e3fb6, 6e6eae6748bd4849aa05bad1e22daf60; no voters: 105f4753f35f455c9ebfdf86ba7c065c
I20260924 03:03:00.750487 7276 leader_election.cc:302] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 065ca39885b54be2bd9fe99c4d2e3fb6, 6e6eae6748bd4849aa05bad1e22daf60; no voters: 105f4753f35f455c9ebfdf86ba7c065c
I20260924 03:03:00.750592 7813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 03:03:00.750622 7812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 03:03:00.750636 7813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 105f4753f35f455c9ebfdf86ba7c065c)
I20260924 03:03:00.750653 7812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 105f4753f35f455c9ebfdf86ba7c065c)
I20260924 03:03:00.750666 7813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:03:00.750674 7812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:03:00.751683 7812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: false } }
I20260924 03:03:00.751682 7813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: false } }
I20260924 03:03:00.751830 7812 leader_election.cc:288] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [CANDIDATE]: Term 2 election: Requested vote from peers 105f4753f35f455c9ebfdf86ba7c065c (127.0.98.62:36379), 065ca39885b54be2bd9fe99c4d2e3fb6 (127.0.98.61:45601)
I20260924 03:03:00.751850 7813 leader_election.cc:288] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [CANDIDATE]: Term 2 election: Requested vote from peers 105f4753f35f455c9ebfdf86ba7c065c (127.0.98.62:36379), 6e6eae6748bd4849aa05bad1e22daf60 (127.0.98.60:40835)
I20260924 03:03:00.752146 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:00.752026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.0.1:54744)
0924 03:03:00.752088 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:00.752118 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 03:03:00.752153 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:00.752044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.0.1:54756)
0924 03:03:00.752098 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:00.752125 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 03:03:00.752228 7300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6e6eae6748bd4849aa05bad1e22daf60"
I20260924 03:03:00.752313 7300 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 065ca39885b54be2bd9fe99c4d2e3fb6 in current term 2: Already voted for candidate 6e6eae6748bd4849aa05bad1e22daf60 in this term.
I20260924 03:03:00.752255 7108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e6eae6748bd4849aa05bad1e22daf60" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6"
I20260924 03:03:00.752524 7108 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e6eae6748bd4849aa05bad1e22daf60 in current term 2: Already voted for candidate 065ca39885b54be2bd9fe99c4d2e3fb6 in this term.
I20260924 03:03:00.752578 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:00.752496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.0.1:54772)
0924 03:03:00.752532 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:00.752555 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 03:03:00.752626 7276 leader_election.cc:334] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 105f4753f35f455c9ebfdf86ba7c065c (127.0.98.62:36379): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
I20260924 03:03:00.752753 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:00.752642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.0.1:54778)
0924 03:03:00.752708 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:00.752733 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 03:03:00.752748 7276 leader_election.cc:302] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e6eae6748bd4849aa05bad1e22daf60; no voters: 065ca39885b54be2bd9fe99c4d2e3fb6, 105f4753f35f455c9ebfdf86ba7c065c
I20260924 03:03:00.752866 7812 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260924 03:03:00.752904 7084 leader_election.cc:334] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 105f4753f35f455c9ebfdf86ba7c065c (127.0.98.62:36379): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
I20260924 03:03:00.753000 7084 leader_election.cc:302] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 065ca39885b54be2bd9fe99c4d2e3fb6; no voters: 105f4753f35f455c9ebfdf86ba7c065c, 6e6eae6748bd4849aa05bad1e22daf60
I20260924 03:03:00.753077 7813 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 03:03:01.241868 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:01.241546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:35591)
0924 03:03:01.241760 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:01.241805 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":129,"threads_started":1}
I20260924 03:03:01.242264 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:01.242049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:35847)
0924 03:03:01.242179 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:01.242217 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":45,"threads_started":1}
I20260924 03:03:01.242376 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:01.241882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:35837)
0924 03:03:01.242289 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:01.242329 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":76,"threads_started":1}
I20260924 03:03:02.242937 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.242735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:58693)
0924 03:03:02.242835 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:02.242875 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":42,"threads_started":1}
I20260924 03:03:02.242955 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.242482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:58755)
0924 03:03:02.242837 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:02.242905 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":131,"threads_started":1}
I20260924 03:03:02.243060 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.242746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:59537)
0924 03:03:02.242984 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:02.243020 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":56,"threads_started":1}
I20260924 03:03:02.376523 7828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:03:02.376637 7828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: false } }
I20260924 03:03:02.376981 7828 leader_election.cc:288] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 105f4753f35f455c9ebfdf86ba7c065c (127.0.98.62:36379), 065ca39885b54be2bd9fe99c4d2e3fb6 (127.0.98.61:45601)
I20260924 03:03:02.377336 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.377137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.0.1:59162)
0924 03:03:02.377258 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:02.377300 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":50,"threads_started":1}
I20260924 03:03:02.377426 7108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e6eae6748bd4849aa05bad1e22daf60" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" is_pre_election: true
I20260924 03:03:02.377557 7108 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e6eae6748bd4849aa05bad1e22daf60 in term 2.
I20260924 03:03:02.377800 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.377691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.0.1:59176)
0924 03:03:02.377739 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:02.377766 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 03:03:02.377741 7276 leader_election.cc:302] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 065ca39885b54be2bd9fe99c4d2e3fb6, 6e6eae6748bd4849aa05bad1e22daf60; no voters:
W20260924 03:03:02.377969 7276 leader_election.cc:334] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 105f4753f35f455c9ebfdf86ba7c065c (127.0.98.62:36379): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
I20260924 03:03:02.377971 7828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 03:03:02.378034 7828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:03:02.378059 7828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 2 FOLLOWER]: Advancing to term 3
I20260924 03:03:02.378923 7828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: false } }
I20260924 03:03:02.379060 7828 leader_election.cc:288] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [CANDIDATE]: Term 3 election: Requested vote from peers 105f4753f35f455c9ebfdf86ba7c065c (127.0.98.62:36379), 065ca39885b54be2bd9fe99c4d2e3fb6 (127.0.98.61:45601)
I20260924 03:03:02.379232 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.379145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.0.1:59182)
0924 03:03:02.379179 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:02.379207 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260924 03:03:02.379350 7108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e6eae6748bd4849aa05bad1e22daf60" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6"
I20260924 03:03:02.379437 7108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 2 FOLLOWER]: Advancing to term 3
I20260924 03:03:02.380184 7108 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e6eae6748bd4849aa05bad1e22daf60 in term 3.
I20260924 03:03:02.380375 7276 leader_election.cc:302] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 065ca39885b54be2bd9fe99c4d2e3fb6, 6e6eae6748bd4849aa05bad1e22daf60; no voters:
I20260924 03:03:02.380457 7828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 03:03:02.380584 7828 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [term 3 LEADER]: Becoming Leader. State: Replica: 6e6eae6748bd4849aa05bad1e22daf60, State: Running, Role: LEADER
I20260924 03:03:02.380621 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.380520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.0.1:59194)
0924 03:03:02.380557 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:02.380592 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260924 03:03:02.380704 7276 leader_election.cc:334] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 105f4753f35f455c9ebfdf86ba7c065c (127.0.98.62:36379): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
I20260924 03:03:02.380730 7828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: false } }
I20260924 03:03:02.381107 7831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e6eae6748bd4849aa05bad1e22daf60. Latest consensus state: current_term: 3 leader_uuid: "6e6eae6748bd4849aa05bad1e22daf60" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: false } } }
I20260924 03:03:02.381210 7831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [sys.catalog]: This master's current role is: LEADER
I20260924 03:03:02.381397 7833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:03:02.381865 7833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:03:02.381927 7833 catalog_manager.cc:1259] Loaded cluster ID: 66173977c15d45668f6c482930c04bf9
I20260924 03:03:02.381946 7833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:03:02.382136 7833 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:03:02.382225 7833 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60: Loaded TSK: 0
I20260924 03:03:02.382371 7833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:03:02.440398 7289 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:03:02.442135 7289 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:03:02.443226 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.443092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.0.1:59216)
0924 03:03:02.443164 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:02.443199 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 03:03:02.443382 7108 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [term 3 FOLLOWER]: Refusing update from remote peer 6e6eae6748bd4849aa05bad1e22daf60: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 03:03:02.443655 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.443518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.0.1:59218)
0924 03:03:02.443559 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:02.443629 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 03:03:02.443718 7828 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260924 03:03:02.443887 7276 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 -> Peer 105f4753f35f455c9ebfdf86ba7c065c (127.0.98.62:36379): Couldn't send request to peer 105f4753f35f455c9ebfdf86ba7c065c. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 03:03:02.444402 7835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e6eae6748bd4849aa05bad1e22daf60. Latest consensus state: current_term: 3 leader_uuid: "6e6eae6748bd4849aa05bad1e22daf60" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: false } } }
I20260924 03:03:02.444552 7835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:03:02.444969 7836 mvcc.cc:205] Tried to move back new op lower bound from 7332736952085266432 to 7332736951832133632. Current Snapshot: MvccSnapshot[applied={T|T < 7332736952085266432}]
I20260924 03:03:02.445003 7828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "6e6eae6748bd4849aa05bad1e22daf60" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: false } } }
I20260924 03:03:02.445087 7828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60 [sys.catalog]: This master's current role is: LEADER
I20260924 03:03:02.445091 7835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "6e6eae6748bd4849aa05bad1e22daf60" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "105f4753f35f455c9ebfdf86ba7c065c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } peers { permanent_uuid: "065ca39885b54be2bd9fe99c4d2e3fb6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45601 } attrs { promote: false } } peers { permanent_uuid: "6e6eae6748bd4849aa05bad1e22daf60" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40835 } attrs { promote: false } } }
I20260924 03:03:02.445178 7835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 065ca39885b54be2bd9fe99c4d2e3fb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:03:02.450145 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.446986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34985 (local address 127.0.0.1:51312)
0924 03:03:02.447164 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:02.447176 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:02.447235 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:02.447379 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:02.447380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:02.447382 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:03:02.447439 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:02.447440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:02.448311 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:02.448312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:02.449147 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:02.449148 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:02.449885 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:02.449886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:02.449947 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:02.449951 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:03:02.450033 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 03:03:02.450074 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260924 03:03:02.450145 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.447075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51312 (local address 127.0.98.2:34985)
0924 03:03:02.447179 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:02.447181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:02.447224 (+ 43us) server_negotiation.cc:407] Connection header received
0924 03:03:02.447291 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:02.447292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:02.447304 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:02.447374 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:03:02.447525 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:02.448230 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:02.449244 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:02.449781 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:02.449884 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:02.449897 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:02.450040 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 03:03:02.450074 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":60,"threads_started":1}
I20260924 03:03:02.450543 7540 tablet_service.cc:1467] Processing CreateTablet for tablet 21de4de81a3f4e4f9381f3f0938690b8 (DEFAULT_TABLE table=test-table [id=5f648eb934904194b51d3968c06df5e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:03:02.450613 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.447078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43013 (local address 127.0.0.1:36022)
0924 03:03:02.447460 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:02.447469 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:02.447510 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:02.447788 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:02.447789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:02.447791 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:03:02.447830 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:02.447832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:02.448466 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:02.448467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:02.449644 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:02.449645 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:02.450389 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:02.450390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:02.450458 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:02.450462 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:03:02.450509 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 03:03:02.450544 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":53,"threads_started":1}
I20260924 03:03:02.450680 7540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21de4de81a3f4e4f9381f3f0938690b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:03:02.450820 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.447368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36022 (local address 127.0.98.3:43013)
0924 03:03:02.447659 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:02.447661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:02.447668 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:03:02.447694 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:02.447695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:02.447705 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:02.447774 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:03:02.447896 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:02.448380 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:02.449737 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:02.450302 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:02.450371 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:02.450382 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:02.450697 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 03:03:02.450760 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":44,"threads_started":1}
I20260924 03:03:02.451043 7676 tablet_service.cc:1467] Processing CreateTablet for tablet 21de4de81a3f4e4f9381f3f0938690b8 (DEFAULT_TABLE table=test-table [id=5f648eb934904194b51d3968c06df5e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:03:02.451157 7676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21de4de81a3f4e4f9381f3f0938690b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:03:02.451431 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.446986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33289 (local address 127.0.0.1:57822)
0924 03:03:02.447228 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:02.447236 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:02.447319 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:02.448466 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:02.448467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:02.448468 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:03:02.448518 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:02.448519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:02.449410 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:02.449411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:02.450476 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:02.450478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:02.451175 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:02.451176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:02.451258 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:02.451263 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:03:02.451327 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 03:03:02.451371 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":20,"thread_start_us":83,"threads_started":1}
I20260924 03:03:02.451478 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.447074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57822 (local address 127.0.98.1:33289)
0924 03:03:02.447179 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:02.447181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:02.447304 (+ 123us) server_negotiation.cc:407] Connection header received
0924 03:03:02.448348 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:02.448350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:02.448361 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:02.448426 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:03:02.448596 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:02.449317 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:02.450561 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:02.451088 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:02.451163 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:02.451175 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:02.451345 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 03:03:02.451394 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":52,"threads_started":1}
I20260924 03:03:02.452064 7846 tablet_bootstrap.cc:492] T 21de4de81a3f4e4f9381f3f0938690b8 P a7ccdcca74be438cb9cd448857cbc55a: Bootstrap starting.
I20260924 03:03:02.452349 7403 tablet_service.cc:1467] Processing CreateTablet for tablet 21de4de81a3f4e4f9381f3f0938690b8 (DEFAULT_TABLE table=test-table [id=5f648eb934904194b51d3968c06df5e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:03:02.452466 7846 tablet_bootstrap.cc:654] T 21de4de81a3f4e4f9381f3f0938690b8 P a7ccdcca74be438cb9cd448857cbc55a: Neither blocks nor log segments found. Creating new log.
I20260924 03:03:02.452474 7403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21de4de81a3f4e4f9381f3f0938690b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:03:02.452642 7847 tablet_bootstrap.cc:492] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8: Bootstrap starting.
I20260924 03:03:02.453020 7847 tablet_bootstrap.cc:654] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8: Neither blocks nor log segments found. Creating new log.
I20260924 03:03:02.453119 7846 tablet_bootstrap.cc:492] T 21de4de81a3f4e4f9381f3f0938690b8 P a7ccdcca74be438cb9cd448857cbc55a: No bootstrap required, opened a new log
I20260924 03:03:02.453166 7846 ts_tablet_manager.cc:1397] T 21de4de81a3f4e4f9381f3f0938690b8 P a7ccdcca74be438cb9cd448857cbc55a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:03:02.453302 7846 raft_consensus.cc:348] T 21de4de81a3f4e4f9381f3f0938690b8 P a7ccdcca74be438cb9cd448857cbc55a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bce5a68994249e7884543c4c7276d3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33289 } } peers { permanent_uuid: "a7ccdcca74be438cb9cd448857cbc55a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34985 } } peers { permanent_uuid: "b5a56c6090d644fb8935dc68bbaeedd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43013 } }
I20260924 03:03:02.453366 7846 raft_consensus.cc:374] T 21de4de81a3f4e4f9381f3f0938690b8 P a7ccdcca74be438cb9cd448857cbc55a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:03:02.453384 7846 raft_consensus.cc:729] T 21de4de81a3f4e4f9381f3f0938690b8 P a7ccdcca74be438cb9cd448857cbc55a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7ccdcca74be438cb9cd448857cbc55a, State: Initialized, Role: FOLLOWER
I20260924 03:03:02.453433 7846 consensus_queue.cc:260] T 21de4de81a3f4e4f9381f3f0938690b8 P a7ccdcca74be438cb9cd448857cbc55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bce5a68994249e7884543c4c7276d3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33289 } } peers { permanent_uuid: "a7ccdcca74be438cb9cd448857cbc55a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34985 } } peers { permanent_uuid: "b5a56c6090d644fb8935dc68bbaeedd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43013 } }
I20260924 03:03:02.453604 7846 ts_tablet_manager.cc:1428] T 21de4de81a3f4e4f9381f3f0938690b8 P a7ccdcca74be438cb9cd448857cbc55a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 03:03:02.453891 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.453757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:41595)
0924 03:03:02.453822 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:02.453857 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 03:03:02.454089 7851 tablet_bootstrap.cc:492] T 21de4de81a3f4e4f9381f3f0938690b8 P 7bce5a68994249e7884543c4c7276d3f: Bootstrap starting.
I20260924 03:03:02.454341 7606 heartbeater.cc:503] Master 127.0.98.60:40835 was elected leader, sending a full tablet report...
I20260924 03:03:02.454406 7851 tablet_bootstrap.cc:654] T 21de4de81a3f4e4f9381f3f0938690b8 P 7bce5a68994249e7884543c4c7276d3f: Neither blocks nor log segments found. Creating new log.
I20260924 03:03:02.454833 7847 tablet_bootstrap.cc:492] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8: No bootstrap required, opened a new log
I20260924 03:03:02.454896 7847 ts_tablet_manager.cc:1397] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:03:02.454977 7851 tablet_bootstrap.cc:492] T 21de4de81a3f4e4f9381f3f0938690b8 P 7bce5a68994249e7884543c4c7276d3f: No bootstrap required, opened a new log
I20260924 03:03:02.455052 7851 ts_tablet_manager.cc:1397] T 21de4de81a3f4e4f9381f3f0938690b8 P 7bce5a68994249e7884543c4c7276d3f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:03:02.455029 7847 raft_consensus.cc:348] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bce5a68994249e7884543c4c7276d3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33289 } } peers { permanent_uuid: "a7ccdcca74be438cb9cd448857cbc55a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34985 } } peers { permanent_uuid: "b5a56c6090d644fb8935dc68bbaeedd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43013 } }
I20260924 03:03:02.455096 7847 raft_consensus.cc:374] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:03:02.455124 7847 raft_consensus.cc:729] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a56c6090d644fb8935dc68bbaeedd8, State: Initialized, Role: FOLLOWER
I20260924 03:03:02.455168 7847 consensus_queue.cc:260] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bce5a68994249e7884543c4c7276d3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33289 } } peers { permanent_uuid: "a7ccdcca74be438cb9cd448857cbc55a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34985 } } peers { permanent_uuid: "b5a56c6090d644fb8935dc68bbaeedd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43013 } }
I20260924 03:03:02.455173 7851 raft_consensus.cc:348] T 21de4de81a3f4e4f9381f3f0938690b8 P 7bce5a68994249e7884543c4c7276d3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bce5a68994249e7884543c4c7276d3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33289 } } peers { permanent_uuid: "a7ccdcca74be438cb9cd448857cbc55a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34985 } } peers { permanent_uuid: "b5a56c6090d644fb8935dc68bbaeedd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43013 } }
I20260924 03:03:02.455243 7851 raft_consensus.cc:374] T 21de4de81a3f4e4f9381f3f0938690b8 P 7bce5a68994249e7884543c4c7276d3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:03:02.455271 7851 raft_consensus.cc:729] T 21de4de81a3f4e4f9381f3f0938690b8 P 7bce5a68994249e7884543c4c7276d3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bce5a68994249e7884543c4c7276d3f, State: Initialized, Role: FOLLOWER
I20260924 03:03:02.455359 7847 ts_tablet_manager.cc:1428] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:02.455319 7851 consensus_queue.cc:260] T 21de4de81a3f4e4f9381f3f0938690b8 P 7bce5a68994249e7884543c4c7276d3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bce5a68994249e7884543c4c7276d3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33289 } } peers { permanent_uuid: "a7ccdcca74be438cb9cd448857cbc55a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34985 } } peers { permanent_uuid: "b5a56c6090d644fb8935dc68bbaeedd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43013 } }
I20260924 03:03:02.455583 7851 ts_tablet_manager.cc:1428] T 21de4de81a3f4e4f9381f3f0938690b8 P 7bce5a68994249e7884543c4c7276d3f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 03:03:02.455804 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.455714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:44697)
0924 03:03:02.455752 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:02.455776 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 03:03:02.455814 7742 heartbeater.cc:503] Master 127.0.98.60:40835 was elected leader, sending a full tablet report...
I20260924 03:03:02.456041 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.455899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:48869)
0924 03:03:02.455975 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:02.456007 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 03:03:02.456064 7469 heartbeater.cc:503] Master 127.0.98.60:40835 was elected leader, sending a full tablet report...
I20260924 03:03:02.456177 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.456069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:48869)
0924 03:03:02.456130 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:02.456154 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 03:03:02.456532 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.456443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:32913)
0924 03:03:02.456479 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:02.456508 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260924 03:03:02.463305 7853 raft_consensus.cc:482] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:03:02.463390 7853 raft_consensus.cc:504] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bce5a68994249e7884543c4c7276d3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33289 } } peers { permanent_uuid: "a7ccdcca74be438cb9cd448857cbc55a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34985 } } peers { permanent_uuid: "b5a56c6090d644fb8935dc68bbaeedd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43013 } }
I20260924 03:03:02.463780 7853 leader_election.cc:288] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bce5a68994249e7884543c4c7276d3f (127.0.98.1:33289), a7ccdcca74be438cb9cd448857cbc55a (127.0.98.2:34985)
I20260924 03:03:02.467638 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.464086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55569 (local address 127.0.98.2:34985)
0924 03:03:02.464126 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:02.464128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:02.464137 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:03:02.464175 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:02.464176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:02.464186 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:02.464237 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:03:02.464408 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:02.465154 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:02.466712 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:02.467254 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:02.467330 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:02.467363 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:02.467502 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 03:03:02.467543 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:03:02.467679 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.464051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56487 (local address 127.0.98.1:33289)
0924 03:03:02.464228 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:02.464230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:02.464385 (+ 155us) server_negotiation.cc:407] Connection header received
0924 03:03:02.464410 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:02.464411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:02.464423 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:02.464477 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:03:02.464577 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:02.465268 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:02.466712 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:02.467254 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:02.467330 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:02.467340 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:02.467537 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 03:03:02.467578 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260924 03:03:02.467660 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.463977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34985 (local address 127.0.98.3:55569)
0924 03:03:02.464030 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:02.464043 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:02.464101 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:02.464244 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:02.464245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:02.464248 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:03:02.464333 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:02.464335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:02.465256 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:02.465257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:02.466617 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:02.466619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:02.467342 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:02.467343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:02.467420 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:02.467423 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:03:02.467482 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 03:03:02.467535 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":47}
I20260924 03:03:02.467671 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.463977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33289 (local address 127.0.98.3:56487)
0924 03:03:02.464281 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:02.464293 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:02.464356 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:02.464482 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:02.464483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:02.464488 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:03:02.464529 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:02.464529 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:02.465352 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:02.465352 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:02.466617 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:02.466619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:02.467365 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:02.467366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:02.467423 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:02.467425 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:03:02.467482 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 03:03:02.467535 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":112,"thread_start_us":87,"threads_started":1}
I20260924 03:03:02.468003 7560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21de4de81a3f4e4f9381f3f0938690b8" candidate_uuid: "b5a56c6090d644fb8935dc68bbaeedd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7ccdcca74be438cb9cd448857cbc55a" is_pre_election: true
I20260924 03:03:02.468091 7560 raft_consensus.cc:2454] T 21de4de81a3f4e4f9381f3f0938690b8 P a7ccdcca74be438cb9cd448857cbc55a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a56c6090d644fb8935dc68bbaeedd8 in term 0.
I20260924 03:03:02.468288 7630 leader_election.cc:302] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7ccdcca74be438cb9cd448857cbc55a, b5a56c6090d644fb8935dc68bbaeedd8; no voters:
I20260924 03:03:02.468420 7853 raft_consensus.cc:2781] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:03:02.468343 7423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21de4de81a3f4e4f9381f3f0938690b8" candidate_uuid: "b5a56c6090d644fb8935dc68bbaeedd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bce5a68994249e7884543c4c7276d3f" is_pre_election: true
I20260924 03:03:02.468458 7853 raft_consensus.cc:482] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:03:02.468479 7853 raft_consensus.cc:3037] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:03:02.468487 7423 raft_consensus.cc:2454] T 21de4de81a3f4e4f9381f3f0938690b8 P 7bce5a68994249e7884543c4c7276d3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a56c6090d644fb8935dc68bbaeedd8 in term 0.
I20260924 03:03:02.469220 7853 raft_consensus.cc:504] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bce5a68994249e7884543c4c7276d3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33289 } } peers { permanent_uuid: "a7ccdcca74be438cb9cd448857cbc55a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34985 } } peers { permanent_uuid: "b5a56c6090d644fb8935dc68bbaeedd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43013 } }
I20260924 03:03:02.469367 7853 leader_election.cc:288] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8 [CANDIDATE]: Term 1 election: Requested vote from peers 7bce5a68994249e7884543c4c7276d3f (127.0.98.1:33289), a7ccdcca74be438cb9cd448857cbc55a (127.0.98.2:34985)
I20260924 03:03:02.469507 7423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21de4de81a3f4e4f9381f3f0938690b8" candidate_uuid: "b5a56c6090d644fb8935dc68bbaeedd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bce5a68994249e7884543c4c7276d3f"
I20260924 03:03:02.469508 7560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21de4de81a3f4e4f9381f3f0938690b8" candidate_uuid: "b5a56c6090d644fb8935dc68bbaeedd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7ccdcca74be438cb9cd448857cbc55a"
I20260924 03:03:02.469579 7560 raft_consensus.cc:3037] T 21de4de81a3f4e4f9381f3f0938690b8 P a7ccdcca74be438cb9cd448857cbc55a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:03:02.469579 7423 raft_consensus.cc:3037] T 21de4de81a3f4e4f9381f3f0938690b8 P 7bce5a68994249e7884543c4c7276d3f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:03:02.470310 7423 raft_consensus.cc:2454] T 21de4de81a3f4e4f9381f3f0938690b8 P 7bce5a68994249e7884543c4c7276d3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a56c6090d644fb8935dc68bbaeedd8 in term 1.
I20260924 03:03:02.470310 7560 raft_consensus.cc:2454] T 21de4de81a3f4e4f9381f3f0938690b8 P a7ccdcca74be438cb9cd448857cbc55a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a56c6090d644fb8935dc68bbaeedd8 in term 1.
I20260924 03:03:02.470525 7632 leader_election.cc:302] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bce5a68994249e7884543c4c7276d3f, b5a56c6090d644fb8935dc68bbaeedd8; no voters:
I20260924 03:03:02.470631 7853 raft_consensus.cc:2781] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:03:02.470768 7853 raft_consensus.cc:686] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8 [term 1 LEADER]: Becoming Leader. State: Replica: b5a56c6090d644fb8935dc68bbaeedd8, State: Running, Role: LEADER
I20260924 03:03:02.470850 7853 consensus_queue.cc:237] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bce5a68994249e7884543c4c7276d3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33289 } } peers { permanent_uuid: "a7ccdcca74be438cb9cd448857cbc55a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34985 } } peers { permanent_uuid: "b5a56c6090d644fb8935dc68bbaeedd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43013 } }
I20260924 03:03:02.471459 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.471148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:38723)
0924 03:03:02.471403 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:02.471432 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239}
I20260924 03:03:02.471495 7288 catalog_manager.cc:5495] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5a56c6090d644fb8935dc68bbaeedd8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b5a56c6090d644fb8935dc68bbaeedd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bce5a68994249e7884543c4c7276d3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7ccdcca74be438cb9cd448857cbc55a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a56c6090d644fb8935dc68bbaeedd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43013 } health_report { overall_health: HEALTHY } } }
I20260924 03:03:02.479928 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.477530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36024 (local address 127.0.98.3:43013)
0924 03:03:02.477568 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:02.477570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:02.477578 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:03:02.477651 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:02.477652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:02.477670 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:02.477758 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:03:02.477888 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:02.478383 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:02.479129 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:02.479331 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:02.479412 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:02.479585 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:03:02.479705 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:03:02.479789 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:02.479831 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 03:03:02.479851 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:03:02.481026 7560 raft_consensus.cc:1261] T 21de4de81a3f4e4f9381f3f0938690b8 P a7ccdcca74be438cb9cd448857cbc55a [term 1 FOLLOWER]: Refusing update from remote peer b5a56c6090d644fb8935dc68bbaeedd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:03:02.481063 7423 raft_consensus.cc:1261] T 21de4de81a3f4e4f9381f3f0938690b8 P 7bce5a68994249e7884543c4c7276d3f [term 1 FOLLOWER]: Refusing update from remote peer b5a56c6090d644fb8935dc68bbaeedd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:03:02.481446 7856 consensus_queue.cc:1035] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7ccdcca74be438cb9cd448857cbc55a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:03:02.481681 7853 consensus_queue.cc:1035] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bce5a68994249e7884543c4c7276d3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:03:02.482369 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.482260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:42083)
0924 03:03:02.482305 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:02.482333 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 03:03:02.482921 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.482771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:53377)
0924 03:03:02.482856 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:02.482887 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 03:03:02.483846 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.483755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:60327)
0924 03:03:02.483797 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:02.483821 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 03:03:02.483870 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.483750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:38083)
0924 03:03:02.483802 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:02.483841 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 03:03:02.484205 7859 mvcc.cc:205] Tried to move back new op lower bound from 7332736952240021504 to 7332736952201175040. Current Snapshot: MvccSnapshot[applied={T|T < 7332736952240021504 or (T in {7332736952240021504})}]
I20260924 03:03:02.484575 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.484460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:44845)
0924 03:03:02.484519 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:02.484551 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 03:03:02.485472 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.485151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:36319)
0924 03:03:02.485412 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:02.485444 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243}
I20260924 03:03:02.485875 7289 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:03:02.485991 7289 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:03:02.487346 7289 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e6eae6748bd4849aa05bad1e22daf60: Sending DeleteTablet for 3 replicas of tablet 21de4de81a3f4e4f9381f3f0938690b8
I20260924 03:03:02.487757 7403 tablet_service.cc:1514] Processing DeleteTablet for tablet 21de4de81a3f4e4f9381f3f0938690b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:03:02 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:57822
I20260924 03:03:02.487807 7540 tablet_service.cc:1514] Processing DeleteTablet for tablet 21de4de81a3f4e4f9381f3f0938690b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:03:02 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:51312
I20260924 03:03:02.487970 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 7245
I20260924 03:03:02.488010 7676 tablet_service.cc:1514] Processing DeleteTablet for tablet 21de4de81a3f4e4f9381f3f0938690b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:03:02 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:36022
I20260924 03:03:02.489127 7872 tablet_replica.cc:330] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8: stopping tablet replica
I20260924 03:03:02.489192 7872 raft_consensus.cc:2229] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:03:02.489259 7872 raft_consensus.cc:2258] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:03:02.489343 7871 tablet_replica.cc:330] T 21de4de81a3f4e4f9381f3f0938690b8 P a7ccdcca74be438cb9cd448857cbc55a: stopping tablet replica
I20260924 03:03:02.489394 7871 raft_consensus.cc:2229] T 21de4de81a3f4e4f9381f3f0938690b8 P a7ccdcca74be438cb9cd448857cbc55a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:03:02.489437 7871 raft_consensus.cc:2258] T 21de4de81a3f4e4f9381f3f0938690b8 P a7ccdcca74be438cb9cd448857cbc55a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:03:02.489758 7871 ts_tablet_manager.cc:1905] T 21de4de81a3f4e4f9381f3f0938690b8 P a7ccdcca74be438cb9cd448857cbc55a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:03:02.491986 7872 ts_tablet_manager.cc:1905] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:03:02.492085 7871 ts_tablet_manager.cc:1918] T 21de4de81a3f4e4f9381f3f0938690b8 P a7ccdcca74be438cb9cd448857cbc55a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:03:02.492206 7871 log.cc:1196] T 21de4de81a3f4e4f9381f3f0938690b8 P a7ccdcca74be438cb9cd448857cbc55a: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-1/wal/wals/21de4de81a3f4e4f9381f3f0938690b8
I20260924 03:03:02.492484 7871 ts_tablet_manager.cc:1939] T 21de4de81a3f4e4f9381f3f0938690b8 P a7ccdcca74be438cb9cd448857cbc55a: Deleting consensus metadata
I20260924 03:03:02.492835 7275 catalog_manager.cc:4841] TS a7ccdcca74be438cb9cd448857cbc55a (127.0.98.2:34985): tablet 21de4de81a3f4e4f9381f3f0938690b8 (table test-table [id=5f648eb934904194b51d3968c06df5e4]) successfully deleted
I20260924 03:03:02.493569 7872 ts_tablet_manager.cc:1918] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:03:02.493628 7872 log.cc:1196] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218948286999-392-0/minicluster-data/ts-2/wal/wals/21de4de81a3f4e4f9381f3f0938690b8
I20260924 03:03:02.493856 7872 ts_tablet_manager.cc:1939] T 21de4de81a3f4e4f9381f3f0938690b8 P b5a56c6090d644fb8935dc68bbaeedd8: Deleting consensus metadata
I20260924 03:03:02.494155 7275 catalog_manager.cc:4841] TS b5a56c6090d644fb8935dc68bbaeedd8 (127.0.98.3:43013): tablet 21de4de81a3f4e4f9381f3f0938690b8 (table test-table [id=5f648eb934904194b51d3968c06df5e4]) successfully deleted
I20260924 03:03:02.495244 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:02.495139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33289 (local address 127.0.0.1:57834)
0924 03:03:02.495183 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:02.495211 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 03:03:02.495270 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 7477
W20260924 03:03:02.495388 7277 catalog_manager.cc:4584] TS 7bce5a68994249e7884543c4c7276d3f (127.0.98.1:33289): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 21de4de81a3f4e4f9381f3f0938690b8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33289: connect: Connection refused (error 111)
I20260924 03:03:02.500435 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 7613
I20260924 03:03:02.506369 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 6964
I20260924 03:03:02.512142 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 7152
2026-09-24T03:03:02Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6416 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:03:02 dist-test-slave-wgsb krb5kdc[7875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:03:02 dist-test-slave-wgsb krb5kdc[7875](info): set up 2 sockets
Sep 24 03:03:02 dist-test-slave-wgsb krb5kdc[7875](info): commencing operation
krb5kdc: starting...
W20260924 03:03:03.576208 392 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:03:03 dist-test-slave-wgsb krb5kdc[7875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:03:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:03:03Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 03:03:03.703898 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33095
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:03:03.785101 7888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:03:03.785346 7888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:03:03.785384 7888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:03:03.786553 7888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:03:03.786623 7888 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 03:03:03.786645 7888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:03:03.786667 7888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:03:03.786695 7888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:03:03.786720 7888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:03:03.788216 7888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33095
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:03:03.788486 7888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:03:03.788719 7888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:03:03.793730 7894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:03.793787 7888 server_base.cc:1034] running on GCE node
W20260924 03:03:03.793720 7893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:03.793720 7896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:03.794234 7888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:03:03.794567 7888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:03:03.795773 7888 hybrid_clock.cc:648] HybridClock initialized: now 1790218983795758 us; error 27 us; skew 500 ppm
Sep 24 03:03:03 dist-test-slave-wgsb krb5kdc[7875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218983, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:03:03.798393 7888 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 03:03:03.798872 7888 webserver.cc:460] Webserver started at http://127.0.98.62:36325/ using document root <none> and password file <none>
I20260924 03:03:03.799137 7888 fs_manager.cc:359] Metadata directory not provided
I20260924 03:03:03.799185 7888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:03:03.799351 7888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:03:03.800472 7888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "22ff55de1be043bd9bee0e8586e561d8"
format_stamp: "Formatted at 2026-09-24 03:03:03 on dist-test-slave-wgsb"
server_key: "e4c8eb582efecef81f7b9118799a2407"
server_key_iv: "eee9dcaaafbe52650a599103c8321e79"
server_key_version: "encryptionkey@0"
I20260924 03:03:03.800827 7888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "22ff55de1be043bd9bee0e8586e561d8"
format_stamp: "Formatted at 2026-09-24 03:03:03 on dist-test-slave-wgsb"
server_key: "e4c8eb582efecef81f7b9118799a2407"
server_key_iv: "eee9dcaaafbe52650a599103c8321e79"
server_key_version: "encryptionkey@0"
I20260924 03:03:03.802284 7888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:03:03.803113 7903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:03.803292 7888 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:03:03.803397 7888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "22ff55de1be043bd9bee0e8586e561d8"
format_stamp: "Formatted at 2026-09-24 03:03:03 on dist-test-slave-wgsb"
server_key: "e4c8eb582efecef81f7b9118799a2407"
server_key_iv: "eee9dcaaafbe52650a599103c8321e79"
server_key_version: "encryptionkey@0"
I20260924 03:03:03.803476 7888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:03:03.820999 7888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:03:03.823738 7888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:03:03.823926 7888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:03:03.829195 7888 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33095
I20260924 03:03:03.829260 7965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33095 every 8 connection(s)
I20260924 03:03:03.829671 7888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:03:03.830346 7966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:03:03.832778 7966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ff55de1be043bd9bee0e8586e561d8: Bootstrap starting.
I20260924 03:03:03.833333 7966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22ff55de1be043bd9bee0e8586e561d8: Neither blocks nor log segments found. Creating new log.
I20260924 03:03:03.833612 7966 log.cc:824] T 00000000000000000000000000000000 P 22ff55de1be043bd9bee0e8586e561d8: Log is configured to *not* fsync() on all Append() calls
I20260924 03:03:03.834301 7966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ff55de1be043bd9bee0e8586e561d8: No bootstrap required, opened a new log
I20260924 03:03:03.836505 7966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22ff55de1be043bd9bee0e8586e561d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ff55de1be043bd9bee0e8586e561d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33095 } }
I20260924 03:03:03.836663 7966 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22ff55de1be043bd9bee0e8586e561d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:03:03.836687 7966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22ff55de1be043bd9bee0e8586e561d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ff55de1be043bd9bee0e8586e561d8, State: Initialized, Role: FOLLOWER
I20260924 03:03:03.836840 7966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22ff55de1be043bd9bee0e8586e561d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ff55de1be043bd9bee0e8586e561d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33095 } }
I20260924 03:03:03.836932 7966 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22ff55de1be043bd9bee0e8586e561d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:03:03.836963 7966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22ff55de1be043bd9bee0e8586e561d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:03:03.836994 7966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22ff55de1be043bd9bee0e8586e561d8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:03:03.837519 7966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22ff55de1be043bd9bee0e8586e561d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ff55de1be043bd9bee0e8586e561d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33095 } }
I20260924 03:03:03.837607 7966 leader_election.cc:302] T 00000000000000000000000000000000 P 22ff55de1be043bd9bee0e8586e561d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22ff55de1be043bd9bee0e8586e561d8; no voters:
I20260924 03:03:03.837744 7966 leader_election.cc:288] T 00000000000000000000000000000000 P 22ff55de1be043bd9bee0e8586e561d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:03:03.837857 7969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22ff55de1be043bd9bee0e8586e561d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:03:03.838016 7966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22ff55de1be043bd9bee0e8586e561d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:03:03.838086 7969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22ff55de1be043bd9bee0e8586e561d8 [term 1 LEADER]: Becoming Leader. State: Replica: 22ff55de1be043bd9bee0e8586e561d8, State: Running, Role: LEADER
I20260924 03:03:03.838243 7969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22ff55de1be043bd9bee0e8586e561d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ff55de1be043bd9bee0e8586e561d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33095 } }
I20260924 03:03:03.838646 7971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22ff55de1be043bd9bee0e8586e561d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22ff55de1be043bd9bee0e8586e561d8. Latest consensus state: current_term: 1 leader_uuid: "22ff55de1be043bd9bee0e8586e561d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ff55de1be043bd9bee0e8586e561d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33095 } } }
I20260924 03:03:03.838797 7971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22ff55de1be043bd9bee0e8586e561d8 [sys.catalog]: This master's current role is: LEADER
I20260924 03:03:03.838893 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 7888
I20260924 03:03:03.839000 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:03:03.839232 392 external_mini_cluster.cc:1428] Setting key cee2c17204d4e4d23551bb3253b00e2d
I20260924 03:03:03.839270 7978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:03:03.840034 7970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22ff55de1be043bd9bee0e8586e561d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22ff55de1be043bd9bee0e8586e561d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ff55de1be043bd9bee0e8586e561d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33095 } } }
I20260924 03:03:03.840265 7970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22ff55de1be043bd9bee0e8586e561d8 [sys.catalog]: This master's current role is: LEADER
I20260924 03:03:03.840121 7978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:03:03.842324 7978 catalog_manager.cc:1347] Generated new cluster ID: ae78efc937874b748391ff495c3ade37
I20260924 03:03:03.842386 7978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:03:03 dist-test-slave-wgsb krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:03:03.849009 7978 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:03:03.849725 7978 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:03:03.855144 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:03.840016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35594 (local address 127.0.98.62:33095)
0924 03:03:03.840462 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:03.840472 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:03.840508 (+ 36us) server_negotiation.cc:407] Connection header received
0924 03:03:03.840642 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:03.840650 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:03.840714 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:03.840870 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:03.841669 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:03.844315 (+ 2646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:03.845196 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:03.845723 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:03.845872 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:03.848572 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:03.848578 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:03.848578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:03:03.848593 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:03.853089 (+ 4496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:03.853551 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:03.853554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:03.853555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:03.853648 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:03.853868 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:03.853869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:03.853870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:03.854109 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:03.854258 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:03.854387 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 03:03:03.854868 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":76,"threads_started":1}
I20260924 03:03:03.855417 7978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22ff55de1be043bd9bee0e8586e561d8: Generated new TSK 0
I20260924 03:03:03.855649 7978 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 03:03:03.891831 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33095
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:03:03.983700 7992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:03:03.983938 7992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:03:03.983978 7992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:03:03.985191 7992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:03:03.985260 7992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:03:03.985298 7992 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:03:03.985357 7992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:03:03.987063 7992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:03:03.987327 7992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:03:03.987565 7992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:03:03.993501 7997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:03.993513 7998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:03.993517 8000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:03.993796 7992 server_base.cc:1034] running on GCE node
I20260924 03:03:03.994084 7992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:03:03.994342 7992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:03:03.995508 7992 hybrid_clock.cc:648] HybridClock initialized: now 1790218983995498 us; error 30 us; skew 500 ppm
Sep 24 03:03:03 dist-test-slave-wgsb krb5kdc[7875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218983, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:03:03.998111 7992 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 03:03:03.998538 7992 webserver.cc:460] Webserver started at http://127.0.98.1:38803/ using document root <none> and password file <none>
I20260924 03:03:03.998770 7992 fs_manager.cc:359] Metadata directory not provided
I20260924 03:03:03.998838 7992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:03:03.998989 7992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:03:04.000157 7992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-0/data/instance:
uuid: "b6c64a32d2274811a9476856840a6e1b"
format_stamp: "Formatted at 2026-09-24 03:03:03 on dist-test-slave-wgsb"
server_key: "146c06744b9ceef79c75067967e6226a"
server_key_iv: "f08e853ec09608b20e6e17ca6cfc4997"
server_key_version: "encryptionkey@0"
I20260924 03:03:04.000514 7992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b6c64a32d2274811a9476856840a6e1b"
format_stamp: "Formatted at 2026-09-24 03:03:03 on dist-test-slave-wgsb"
server_key: "146c06744b9ceef79c75067967e6226a"
server_key_iv: "f08e853ec09608b20e6e17ca6cfc4997"
server_key_version: "encryptionkey@0"
I20260924 03:03:04.001856 7992 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:03:04.002804 8007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:04.003017 7992 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:03:04.003127 7992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "b6c64a32d2274811a9476856840a6e1b"
format_stamp: "Formatted at 2026-09-24 03:03:03 on dist-test-slave-wgsb"
server_key: "146c06744b9ceef79c75067967e6226a"
server_key_iv: "f08e853ec09608b20e6e17ca6cfc4997"
server_key_version: "encryptionkey@0"
I20260924 03:03:04.003209 7992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:03:04.037950 7992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:03:04.040743 7992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:03:04.040948 7992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:03:04.041779 7992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:03:04.041836 7992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:04.041888 7992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:03:04.041914 7992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:04.049510 7992 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35847
I20260924 03:03:04.049705 8122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35847 every 8 connection(s)
I20260924 03:03:04.049911 7992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 03:03:04 dist-test-slave-wgsb krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218983, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:03:04.053988 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.042213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57125 (local address 127.0.98.62:33095)
0924 03:03:04.042306 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:04.042309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:04.043073 (+ 764us) server_negotiation.cc:407] Connection header received
0924 03:03:04.043765 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:04.043766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:04.043784 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:04.043854 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:04.045095 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.046008 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.047820 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.048171 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.048265 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.051035 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:04.051036 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:04.051037 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:03:04.051046 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:04.052606 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:04.053029 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:04.053030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:04.053031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:04.053092 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:04.053338 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:04.053339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:04.053339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:04.053456 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:04.053541 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:04.053797 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 03:03:04.053896 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 03:03:04.054179 8016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.042559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33095 (local address 127.0.98.1:57125)
0924 03:03:04.042881 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:04.042926 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:04.043646 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:04.043941 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:04.043945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:04.044663 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:04.044983 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.044991 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.046139 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.046140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.047749 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.047751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.048265 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.048266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.048429 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.049103 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:04.049111 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:04.050938 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:03:04.052710 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:03:04.052717 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:03:04.052719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:03:04.052939 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:03:04.053159 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:03:04.053160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:03:04.053160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:03:04.053272 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:03:04.053524 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:03:04.053530 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:03:04.053670 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 03:03:04.053776 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 03:03:04.053928 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":91,"threads_started":1}
I20260924 03:03:04.054702 8123 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33095
I20260924 03:03:04.054795 8123 heartbeater.cc:464] Registering TS with master...
I20260924 03:03:04.054996 8123 heartbeater.cc:511] Master 127.0.98.62:33095 requested a full tablet report, sending...
I20260924 03:03:04.055579 7920 ts_manager.cc:194] Registered new tserver with Master: b6c64a32d2274811a9476856840a6e1b (127.0.98.1:35847)
I20260924 03:03:04.056479 7920 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57125
I20260924 03:03:04.058027 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 7992
I20260924 03:03:04.058136 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:03:04.058339 392 external_mini_cluster.cc:1428] Setting key 3e462c5e61b6c4ddb65f2c534dcc0840
I20260924 03:03:04.064285 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.056022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52837 (local address 127.0.98.62:33095)
0924 03:03:04.056065 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:04.056067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:04.056142 (+ 75us) server_negotiation.cc:407] Connection header received
0924 03:03:04.056192 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:04.056193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:04.056202 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:04.056251 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:04.056691 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.057561 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.058602 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.058856 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.059018 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.060600 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:04.060602 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:04.060603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:03:04.060611 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:04.062577 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:04.063195 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:04.063196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:04.063197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:04.063279 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:04.063610 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:04.063630 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:04.063631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:04.063787 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:04.063846 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:04.064149 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 03:03:04.064220 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 03:03:04.064316 8129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.055924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33095 (local address 127.0.98.1:52837)
0924 03:03:04.056069 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:04.056081 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:04.056138 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:04.056256 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:04.056256 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:04.056439 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:04.056501 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.056503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.057682 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.057683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.058475 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.058479 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.059085 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.059086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.059164 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.059779 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:04.059785 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:04.060456 (+ 671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:03:04.062701 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:03:04.062703 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:03:04.062703 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:03:04.063006 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:03:04.063366 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:03:04.063367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:03:04.063367 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:03:04.063493 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:03:04.063864 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:03:04.063866 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:03:04.063930 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 03:03:04.064026 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 03:03:04.064169 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 03:03:04.088214 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33095
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:03:04.170576 8132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:03:04.170835 8132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:03:04.170876 8132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:03:04.172255 8132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:03:04.172328 8132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:03:04.172363 8132 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:03:04.172405 8132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:03:04.173936 8132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:03:04.174233 8132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:03:04.174480 8132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:03:04.180027 8138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:04.180042 8132 server_base.cc:1034] running on GCE node
W20260924 03:03:04.179998 8137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:04.179988 8140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:04.180462 8132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:03:04.180728 8132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:03:04.181908 8132 hybrid_clock.cc:648] HybridClock initialized: now 1790218984181894 us; error 24 us; skew 500 ppm
Sep 24 03:03:04 dist-test-slave-wgsb krb5kdc[7875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218984, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:03:04.184571 8132 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 03:03:04.185065 8132 webserver.cc:460] Webserver started at http://127.0.98.2:45925/ using document root <none> and password file <none>
I20260924 03:03:04.185302 8132 fs_manager.cc:359] Metadata directory not provided
I20260924 03:03:04.185369 8132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:03:04.185527 8132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:03:04.186604 8132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-1/data/instance:
uuid: "eb6bbbb5b66f46edbffd1964a617489a"
format_stamp: "Formatted at 2026-09-24 03:03:04 on dist-test-slave-wgsb"
server_key: "1034501fbc31b3d956ffea2ad50bc478"
server_key_iv: "850ac93bcbc84ab61d25cfbcc76cf2cb"
server_key_version: "encryptionkey@0"
I20260924 03:03:04.186960 8132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-1/wal/instance:
uuid: "eb6bbbb5b66f46edbffd1964a617489a"
format_stamp: "Formatted at 2026-09-24 03:03:04 on dist-test-slave-wgsb"
server_key: "1034501fbc31b3d956ffea2ad50bc478"
server_key_iv: "850ac93bcbc84ab61d25cfbcc76cf2cb"
server_key_version: "encryptionkey@0"
I20260924 03:03:04.188345 8132 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:03:04.189170 8147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:04.189337 8132 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:03:04.189435 8132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "eb6bbbb5b66f46edbffd1964a617489a"
format_stamp: "Formatted at 2026-09-24 03:03:04 on dist-test-slave-wgsb"
server_key: "1034501fbc31b3d956ffea2ad50bc478"
server_key_iv: "850ac93bcbc84ab61d25cfbcc76cf2cb"
server_key_version: "encryptionkey@0"
I20260924 03:03:04.189517 8132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:03:04.212069 8132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:03:04.214283 8132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:03:04.214455 8132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:03:04.215026 8132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:03:04.215072 8132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:04.215097 8132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:03:04.215145 8132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:03:04 dist-test-slave-wgsb krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218984, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:03:04.222525 8132 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39853
I20260924 03:03:04.222623 8262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39853 every 8 connection(s)
I20260924 03:03:04.222965 8132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:03:04.223448 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 8132
I20260924 03:03:04.223565 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:03:04.223816 392 external_mini_cluster.cc:1428] Setting key 3a1e7a35961b99f37cd5c000ff21ee52
I20260924 03:03:04.225513 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.215568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59461 (local address 127.0.98.62:33095)
0924 03:03:04.215643 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:04.215646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:04.215817 (+ 171us) server_negotiation.cc:407] Connection header received
0924 03:03:04.216318 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:04.216320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:04.216334 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:04.216420 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:04.217105 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.217957 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.219056 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.219411 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.219491 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.222396 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:04.222398 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:04.222398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:03:04.222407 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:04.224098 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:04.224520 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:04.224521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:04.224521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:04.224587 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:04.224865 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:04.224866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:04.224866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:04.224985 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:04.225061 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:04.225329 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 03:03:04.225423 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 03:03:04.225723 8156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.215493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33095 (local address 127.0.98.2:59461)
0924 03:03:04.215684 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:04.215724 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:04.216215 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:04.216515 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:04.216519 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:04.216834 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:04.217020 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.217024 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.218078 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.218079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.218939 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.218941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.219541 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.219543 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.219716 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.220318 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:04.220328 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:04.222306 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:03:04.224214 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:03:04.224218 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:03:04.224220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:03:04.224439 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:03:04.224661 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:03:04.224662 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:03:04.224662 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:03:04.224801 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:03:04.225053 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:03:04.225059 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:03:04.225198 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 03:03:04.225307 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 03:03:04.225469 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":104,"threads_started":1}
I20260924 03:03:04.226255 8263 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33095
I20260924 03:03:04.226341 8263 heartbeater.cc:464] Registering TS with master...
I20260924 03:03:04.226531 8263 heartbeater.cc:511] Master 127.0.98.62:33095 requested a full tablet report, sending...
I20260924 03:03:04.226908 7919 ts_manager.cc:194] Registered new tserver with Master: eb6bbbb5b66f46edbffd1964a617489a (127.0.98.2:39853)
I20260924 03:03:04.227403 7919 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59461
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 03:03:04.233777 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.227380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41795 (local address 127.0.98.62:33095)
0924 03:03:04.227538 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:04.227540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:04.227727 (+ 187us) server_negotiation.cc:407] Connection header received
0924 03:03:04.227817 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:04.227818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:04.227828 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:04.227879 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:04.228274 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.228720 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.229877 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.230063 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.230157 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.231155 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:04.231156 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:04.231156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:03:04.231162 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:04.232541 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:04.232936 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:04.232937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:04.232937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:04.233030 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:04.233305 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:04.233306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:04.233306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:04.233446 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:04.233500 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:04.233649 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 03:03:04.233715 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260924 03:03:04.233786 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.227321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33095 (local address 127.0.98.2:41795)
0924 03:03:04.227670 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:04.227681 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:04.227752 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:04.227895 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:04.227896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:04.228106 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:04.228186 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.228188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.228820 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.228821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.229786 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.229789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.230166 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.230167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.230256 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.230682 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:04.230687 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:04.231072 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:03:04.232628 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:03:04.232629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:03:04.232629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:03:04.232831 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:03:04.233165 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:03:04.233166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:03:04.233166 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:03:04.233235 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:03:04.233506 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:03:04.233506 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:03:04.233579 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 03:03:04.233632 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 03:03:04.233711 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":9,"thread_start_us":73,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 03:03:04.250952 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33095
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:03:04.333833 8272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:03:04.334043 8272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:03:04.334066 8272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:03:04.335250 8272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:03:04.335292 8272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:03:04.335314 8272 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:03:04.335330 8272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:03:04.336894 8272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:03:04.337131 8272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:03:04.337375 8272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:03:04.342957 8277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:04.343140 8280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:04.343252 8278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:04.343225 8272 server_base.cc:1034] running on GCE node
I20260924 03:03:04.343415 8272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:03:04.343664 8272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:03:04.344833 8272 hybrid_clock.cc:648] HybridClock initialized: now 1790218984344797 us; error 55 us; skew 500 ppm
Sep 24 03:03:04 dist-test-slave-wgsb krb5kdc[7875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218984, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:03:04.347424 8272 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 03:03:04.347915 8272 webserver.cc:460] Webserver started at http://127.0.98.3:44965/ using document root <none> and password file <none>
I20260924 03:03:04.348150 8272 fs_manager.cc:359] Metadata directory not provided
I20260924 03:03:04.348217 8272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:03:04.348373 8272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:03:04.349510 8272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-2/data/instance:
uuid: "1407f59074514c53a31262544bbf537f"
format_stamp: "Formatted at 2026-09-24 03:03:04 on dist-test-slave-wgsb"
server_key: "152950e8ac3003ee85918fdfddcc5f5d"
server_key_iv: "c28ced4e9438fc81a38ed1e58aa2d602"
server_key_version: "encryptionkey@0"
I20260924 03:03:04.349877 8272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1407f59074514c53a31262544bbf537f"
format_stamp: "Formatted at 2026-09-24 03:03:04 on dist-test-slave-wgsb"
server_key: "152950e8ac3003ee85918fdfddcc5f5d"
server_key_iv: "c28ced4e9438fc81a38ed1e58aa2d602"
server_key_version: "encryptionkey@0"
I20260924 03:03:04.351248 8272 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:03:04.352094 8287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:04.352290 8272 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 03:03:04.352406 8272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "1407f59074514c53a31262544bbf537f"
format_stamp: "Formatted at 2026-09-24 03:03:04 on dist-test-slave-wgsb"
server_key: "152950e8ac3003ee85918fdfddcc5f5d"
server_key_iv: "c28ced4e9438fc81a38ed1e58aa2d602"
server_key_version: "encryptionkey@0"
I20260924 03:03:04.352483 8272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:03:04.369228 8272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:03:04.371500 8272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:03:04.371711 8272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:03:04.372263 8272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:03:04.372329 8272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:04.372375 8272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:03:04.372402 8272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:03:04 dist-test-slave-wgsb krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218984, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:03:04.379619 8272 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39147
I20260924 03:03:04.379684 8402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39147 every 8 connection(s)
I20260924 03:03:04.380040 8272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:03:04.382478 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.372821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46667 (local address 127.0.98.62:33095)
0924 03:03:04.372885 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:04.372888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:04.373072 (+ 184us) server_negotiation.cc:407] Connection header received
0924 03:03:04.373561 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:04.373563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:04.373577 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:04.373650 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:04.374504 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.375344 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.376666 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.377019 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.377096 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.379634 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:04.379636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:04.379636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:03:04.379644 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:04.381130 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:04.381539 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:04.381541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:04.381541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:04.381598 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:04.381842 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:04.381843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:04.381844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:04.381962 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:04.382048 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:04.382306 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 03:03:04.382386 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 03:03:04.382673 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.372795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33095 (local address 127.0.98.3:46667)
0924 03:03:04.372939 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:04.372978 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:04.373466 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:04.373730 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:04.373734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:04.374100 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:04.374401 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.374408 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.375449 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.375451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.376585 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.376588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.377131 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.377132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.377276 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.377856 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:04.377864 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:04.379504 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:03:04.381225 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:03:04.381232 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:03:04.381234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:03:04.381452 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:03:04.381662 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:03:04.381663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:03:04.381664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:03:04.381774 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:03:04.382029 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:03:04.382034 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:03:04.382175 (+ 141us) client_negotiation.cc:769] Sending connection context
0924 03:03:04.382284 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 03:03:04.382440 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
I20260924 03:03:04.383250 8403 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33095
I20260924 03:03:04.383358 8403 heartbeater.cc:464] Registering TS with master...
I20260924 03:03:04.383556 8403 heartbeater.cc:511] Master 127.0.98.62:33095 requested a full tablet report, sending...
I20260924 03:03:04.384034 7920 ts_manager.cc:194] Registered new tserver with Master: 1407f59074514c53a31262544bbf537f (127.0.98.3:39147)
I20260924 03:03:04.384639 7920 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46667
I20260924 03:03:04.385879 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 8272
I20260924 03:03:04.385967 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:03:04.386158 392 external_mini_cluster.cc:1428] Setting key 3f037ac2861a29c4afbba5f5f7e67577
I20260924 03:03:04.387137 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:03:04.391038 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.384345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53973 (local address 127.0.98.62:33095)
0924 03:03:04.384385 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:04.384387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:04.384445 (+ 58us) server_negotiation.cc:407] Connection header received
0924 03:03:04.384515 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:04.384516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:04.384526 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:04.384591 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:04.384887 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.385665 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.386704 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.386965 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.387037 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.388079 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:04.388080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:04.388081 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:03:04.388087 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:04.389701 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:04.390112 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:04.390113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:04.390114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:04.390177 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:04.390488 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:04.390489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:04.390489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:04.390607 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:04.390691 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:04.390899 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 03:03:04.390978 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 03:03:04.391371 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.384274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33095 (local address 127.0.98.3:53973)
0924 03:03:04.384389 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:04.384398 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:04.384457 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:04.384593 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:04.384594 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:04.384728 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:04.384816 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.384818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.385781 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.385782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.386624 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.386627 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.387042 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.387043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.387128 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.387516 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:04.387521 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:04.387970 (+ 449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:03:04.389819 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:03:04.389820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:03:04.389820 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:03:04.390018 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:03:04.390348 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:03:04.390349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:03:04.390349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:03:04.390406 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:03:04.390682 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:03:04.390683 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:03:04.390742 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 03:03:04.391180 (+ 438us) client_negotiation.cc:240] Negotiation successful
0924 03:03:04.391300 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":69,"threads_started":1}
Sep 24 03:03:04 dist-test-slave-wgsb krb5kdc[7875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218984, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 03:03:04 dist-test-slave-wgsb krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:04 dist-test-slave-wgsb krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 03:03:04.409626 7987 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:04.403475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35604 (local address 127.0.98.62:33095)
0924 03:03:04.403554 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:04.403556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:04.403565 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:03:04.403588 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:04.403650 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:04.403662 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:04.403730 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:04.404346 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.404818 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.405732 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.405905 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.405955 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.409520 (+ 3565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35604: BlockingRecv error: recv got EOF from 127.0.0.1:35604 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 03:03:04 dist-test-slave-wgsb krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218984, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:03:04.417615 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.410491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35606 (local address 127.0.98.62:33095)
0924 03:03:04.410529 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:04.410531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:04.410670 (+ 139us) server_negotiation.cc:407] Connection header received
0924 03:03:04.410704 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:04.410705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:04.410714 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:04.410754 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:04.411288 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.411873 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.412630 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.412807 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.412864 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.415166 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:04.415168 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:04.415168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:03:04.415175 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:04.416598 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:04.416928 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:04.416928 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:04.416929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:04.416976 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:04.417185 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:04.417186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:04.417186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:04.417283 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:04.417356 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:04.417489 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 03:03:04.417548 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:03:04.419454 7920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:03:04.421133 7920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:03:04.427467 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.423363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35847 (local address 127.0.0.1:59238)
0924 03:03:04.423482 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:04.423521 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:04.423621 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:04.423945 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:04.423949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:04.423952 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:03:04.424025 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.424027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.425380 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.425381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.426437 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.426439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.427127 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.427128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.427247 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.427265 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 03:03:04.427336 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 03:03:04.427384 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":77,"thread_start_us":51,"threads_started":1}
I20260924 03:03:04.427675 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.423530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59238 (local address 127.0.98.1:35847)
0924 03:03:04.423760 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:04.423766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:04.423776 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:03:04.423827 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:04.423829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:04.423845 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:04.423937 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:03:04.424113 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.425287 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.426522 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.427045 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.427116 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.427238 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:04.427395 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 03:03:04.427433 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":126,"threads_started":1}
I20260924 03:03:04.427867 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.423559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39147 (local address 127.0.0.1:40402)
0924 03:03:04.423890 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:04.423900 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:04.423944 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:04.424401 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:04.424402 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:04.424404 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:03:04.424449 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.424451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.425344 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.425345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.426729 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.426731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.427627 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.427628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.427704 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.427708 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:03:04.427766 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 03:03:04.427804 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":79,"thread_start_us":110,"threads_started":1}
I20260924 03:03:04.428063 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.423732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40402 (local address 127.0.98.3:39147)
0924 03:03:04.424158 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:04.424161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:04.424204 (+ 43us) server_negotiation.cc:407] Connection header received
0924 03:03:04.424242 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:04.424244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:04.424260 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:04.424334 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:03:04.424510 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.425249 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.426817 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.427531 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.427633 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.427708 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:04.427910 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 03:03:04.427959 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":93,"threads_started":1}
I20260924 03:03:04.428417 8057 tablet_service.cc:1467] Processing CreateTablet for tablet 922235789a5545a592e23d07312ffa89 (DEFAULT_TABLE table=test-table [id=195a54eda1d146fabcb57305347b352a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:03:04.428665 8337 tablet_service.cc:1467] Processing CreateTablet for tablet 922235789a5545a592e23d07312ffa89 (DEFAULT_TABLE table=test-table [id=195a54eda1d146fabcb57305347b352a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:03:04.428804 8057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 922235789a5545a592e23d07312ffa89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:03:04.428956 8337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 922235789a5545a592e23d07312ffa89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:03:04.432541 8431 tablet_bootstrap.cc:492] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f: Bootstrap starting.
I20260924 03:03:04.432665 8432 tablet_bootstrap.cc:492] T 922235789a5545a592e23d07312ffa89 P b6c64a32d2274811a9476856840a6e1b: Bootstrap starting.
I20260924 03:03:04.433271 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.423771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39853 (local address 127.0.0.1:51912)
0924 03:03:04.424011 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:04.424018 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:04.424062 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:04.424323 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:04.424324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:04.424326 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:03:04.424372 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.424374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.425555 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.425556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.426951 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.426952 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.433005 (+ 6053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.433006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.433082 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.433088 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 03:03:04.433148 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 03:03:04.433196 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":2,"thread_start_us":71,"threads_started":1}
I20260924 03:03:04.433276 8431 tablet_bootstrap.cc:654] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f: Neither blocks nor log segments found. Creating new log.
I20260924 03:03:04.433276 8432 tablet_bootstrap.cc:654] T 922235789a5545a592e23d07312ffa89 P b6c64a32d2274811a9476856840a6e1b: Neither blocks nor log segments found. Creating new log.
I20260924 03:03:04.433293 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.423882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51912 (local address 127.0.98.2:39853)
0924 03:03:04.424041 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:04.424045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:04.424060 (+ 15us) server_negotiation.cc:407] Connection header received
0924 03:03:04.424180 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:04.424183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:04.424199 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:04.424265 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:03:04.424446 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.425475 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.427033 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.432906 (+ 5873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.433003 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.433080 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:04.433163 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 03:03:04.433205 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":65,"threads_started":1}
I20260924 03:03:04.433604 8431 log.cc:824] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f: Log is configured to *not* fsync() on all Append() calls
I20260924 03:03:04.433604 8432 log.cc:824] T 922235789a5545a592e23d07312ffa89 P b6c64a32d2274811a9476856840a6e1b: Log is configured to *not* fsync() on all Append() calls
I20260924 03:03:04.434187 8197 tablet_service.cc:1467] Processing CreateTablet for tablet 922235789a5545a592e23d07312ffa89 (DEFAULT_TABLE table=test-table [id=195a54eda1d146fabcb57305347b352a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:03:04.434393 8431 tablet_bootstrap.cc:492] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f: No bootstrap required, opened a new log
I20260924 03:03:04.434469 8431 ts_tablet_manager.cc:1397] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:03:04.434520 8432 tablet_bootstrap.cc:492] T 922235789a5545a592e23d07312ffa89 P b6c64a32d2274811a9476856840a6e1b: No bootstrap required, opened a new log
I20260924 03:03:04.434544 8197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 922235789a5545a592e23d07312ffa89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:03:04.434679 8432 ts_tablet_manager.cc:1397] T 922235789a5545a592e23d07312ffa89 P b6c64a32d2274811a9476856840a6e1b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:03:04.436709 8432 raft_consensus.cc:348] T 922235789a5545a592e23d07312ffa89 P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } }
I20260924 03:03:04.436713 8431 raft_consensus.cc:348] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } }
I20260924 03:03:04.436888 8431 raft_consensus.cc:374] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:03:04.436888 8432 raft_consensus.cc:374] T 922235789a5545a592e23d07312ffa89 P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:03:04.436926 8432 raft_consensus.cc:729] T 922235789a5545a592e23d07312ffa89 P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6c64a32d2274811a9476856840a6e1b, State: Initialized, Role: FOLLOWER
I20260924 03:03:04.436926 8431 raft_consensus.cc:729] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1407f59074514c53a31262544bbf537f, State: Initialized, Role: FOLLOWER
I20260924 03:03:04.437078 8432 consensus_queue.cc:260] T 922235789a5545a592e23d07312ffa89 P b6c64a32d2274811a9476856840a6e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } }
I20260924 03:03:04.437078 8431 consensus_queue.cc:260] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } }
I20260924 03:03:04.437348 8432 ts_tablet_manager.cc:1428] T 922235789a5545a592e23d07312ffa89 P b6c64a32d2274811a9476856840a6e1b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:03:04.437387 8123 heartbeater.cc:503] Master 127.0.98.62:33095 was elected leader, sending a full tablet report...
I20260924 03:03:04.437527 8403 heartbeater.cc:503] Master 127.0.98.62:33095 was elected leader, sending a full tablet report...
I20260924 03:03:04.437736 8437 tablet_bootstrap.cc:492] T 922235789a5545a592e23d07312ffa89 P eb6bbbb5b66f46edbffd1964a617489a: Bootstrap starting.
I20260924 03:03:04.437348 8431 ts_tablet_manager.cc:1428] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:03:04.438390 8437 tablet_bootstrap.cc:654] T 922235789a5545a592e23d07312ffa89 P eb6bbbb5b66f46edbffd1964a617489a: Neither blocks nor log segments found. Creating new log.
I20260924 03:03:04.438827 8437 log.cc:824] T 922235789a5545a592e23d07312ffa89 P eb6bbbb5b66f46edbffd1964a617489a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:03:04.439504 8437 tablet_bootstrap.cc:492] T 922235789a5545a592e23d07312ffa89 P eb6bbbb5b66f46edbffd1964a617489a: No bootstrap required, opened a new log
I20260924 03:03:04.439558 8437 ts_tablet_manager.cc:1397] T 922235789a5545a592e23d07312ffa89 P eb6bbbb5b66f46edbffd1964a617489a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:03:04.440986 8437 raft_consensus.cc:348] T 922235789a5545a592e23d07312ffa89 P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } }
I20260924 03:03:04.441112 8437 raft_consensus.cc:374] T 922235789a5545a592e23d07312ffa89 P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:03:04.441138 8437 raft_consensus.cc:729] T 922235789a5545a592e23d07312ffa89 P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6bbbb5b66f46edbffd1964a617489a, State: Initialized, Role: FOLLOWER
I20260924 03:03:04.441252 8437 consensus_queue.cc:260] T 922235789a5545a592e23d07312ffa89 P eb6bbbb5b66f46edbffd1964a617489a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } }
I20260924 03:03:04.441474 8437 ts_tablet_manager.cc:1428] T 922235789a5545a592e23d07312ffa89 P eb6bbbb5b66f46edbffd1964a617489a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 03:03:04.441543 8263 heartbeater.cc:503] Master 127.0.98.62:33095 was elected leader, sending a full tablet report...
W20260924 03:03:04.473621 8264 tablet.cc:2369] T 922235789a5545a592e23d07312ffa89 P eb6bbbb5b66f46edbffd1964a617489a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:03:04.550666 8124 tablet.cc:2369] T 922235789a5545a592e23d07312ffa89 P b6c64a32d2274811a9476856840a6e1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:03:04.560910 8435 raft_consensus.cc:482] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:03:04.561125 8435 raft_consensus.cc:504] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } }
I20260924 03:03:04.561414 8435 leader_election.cc:288] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6c64a32d2274811a9476856840a6e1b (127.0.98.1:35847), eb6bbbb5b66f46edbffd1964a617489a (127.0.98.2:39853)
I20260924 03:03:04.565657 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.561921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60609 (local address 127.0.98.2:39853)
0924 03:03:04.561992 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:04.561995 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:04.562126 (+ 131us) server_negotiation.cc:407] Connection header received
0924 03:03:04.562179 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:04.562180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:04.562190 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:04.562255 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:03:04.562387 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.563091 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.564688 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.565250 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.565333 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.565345 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:04.565549 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 03:03:04.565587 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 03:03:04.565645 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.561872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48791 (local address 127.0.98.1:35847)
0924 03:03:04.561933 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:04.561935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:04.561945 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:03:04.561975 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:04.561976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:04.561987 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:04.562053 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:03:04.562265 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.563088 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.564688 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.565277 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.565339 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.565347 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:04.565524 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 03:03:04.565562 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 03:03:04.565678 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.561792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39853 (local address 127.0.98.3:60609)
0924 03:03:04.562045 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:04.562057 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:04.562119 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:04.562258 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:04.562258 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:04.562260 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:03:04.562329 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.562330 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.563194 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.563195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.564593 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.564595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.565358 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.565358 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.565441 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.565446 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:03:04.565508 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 03:03:04.565564 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":55,"thread_start_us":97,"threads_started":1}
I20260924 03:03:04.565687 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.561654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35847 (local address 127.0.98.3:48791)
0924 03:03:04.561719 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:04.561733 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:04.561781 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:04.562109 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:04.562110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:04.562112 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:03:04.562194 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.562196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.563194 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.563195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.564593 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.564595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.565357 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.565358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.565441 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.565452 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 03:03:04.565507 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 03:03:04.565560 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":78}
I20260924 03:03:04.566139 8217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922235789a5545a592e23d07312ffa89" candidate_uuid: "1407f59074514c53a31262544bbf537f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6bbbb5b66f46edbffd1964a617489a" is_pre_election: true
I20260924 03:03:04.566191 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922235789a5545a592e23d07312ffa89" candidate_uuid: "1407f59074514c53a31262544bbf537f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6c64a32d2274811a9476856840a6e1b" is_pre_election: true
I20260924 03:03:04.566277 8217 raft_consensus.cc:2454] T 922235789a5545a592e23d07312ffa89 P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1407f59074514c53a31262544bbf537f in term 0.
I20260924 03:03:04.566361 8062 raft_consensus.cc:2454] T 922235789a5545a592e23d07312ffa89 P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1407f59074514c53a31262544bbf537f in term 0.
I20260924 03:03:04.566522 8291 leader_election.cc:302] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1407f59074514c53a31262544bbf537f, eb6bbbb5b66f46edbffd1964a617489a; no voters:
I20260924 03:03:04.566655 8435 raft_consensus.cc:2781] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:03:04.566708 8435 raft_consensus.cc:482] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:03:04.566736 8435 raft_consensus.cc:3037] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:03:04.567934 8435 raft_consensus.cc:504] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } }
I20260924 03:03:04.568079 8435 leader_election.cc:288] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f [CANDIDATE]: Term 1 election: Requested vote from peers b6c64a32d2274811a9476856840a6e1b (127.0.98.1:35847), eb6bbbb5b66f46edbffd1964a617489a (127.0.98.2:39853)
I20260924 03:03:04.568240 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922235789a5545a592e23d07312ffa89" candidate_uuid: "1407f59074514c53a31262544bbf537f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6c64a32d2274811a9476856840a6e1b"
I20260924 03:03:04.568307 8217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922235789a5545a592e23d07312ffa89" candidate_uuid: "1407f59074514c53a31262544bbf537f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6bbbb5b66f46edbffd1964a617489a"
I20260924 03:03:04.568351 8062 raft_consensus.cc:3037] T 922235789a5545a592e23d07312ffa89 P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:03:04.568406 8217 raft_consensus.cc:3037] T 922235789a5545a592e23d07312ffa89 P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:03:04.569186 8217 raft_consensus.cc:2454] T 922235789a5545a592e23d07312ffa89 P eb6bbbb5b66f46edbffd1964a617489a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1407f59074514c53a31262544bbf537f in term 1.
I20260924 03:03:04.569187 8062 raft_consensus.cc:2454] T 922235789a5545a592e23d07312ffa89 P b6c64a32d2274811a9476856840a6e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1407f59074514c53a31262544bbf537f in term 1.
I20260924 03:03:04.569344 8291 leader_election.cc:302] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1407f59074514c53a31262544bbf537f, b6c64a32d2274811a9476856840a6e1b; no voters:
I20260924 03:03:04.569450 8435 raft_consensus.cc:2781] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:03:04.569625 8435 raft_consensus.cc:686] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f [term 1 LEADER]: Becoming Leader. State: Replica: 1407f59074514c53a31262544bbf537f, State: Running, Role: LEADER
I20260924 03:03:04.569726 8435 consensus_queue.cc:237] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } }
I20260924 03:03:04.570582 7920 catalog_manager.cc:5495] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1407f59074514c53a31262544bbf537f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1407f59074514c53a31262544bbf537f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } health_report { overall_health: UNKNOWN } } }
I20260924 03:03:04.608255 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.605430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40414 (local address 127.0.98.3:39147)
0924 03:03:04.605483 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:04.605485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:04.605506 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:03:04.605561 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:04.605562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:04.605581 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:04.605670 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:03:04.605832 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.606487 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.607396 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.607660 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.607722 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.607904 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:03:04.608026 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:03:04.608124 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:04.608143 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 03:03:04.608165 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 03:03:04.610229 8217 raft_consensus.cc:1261] T 922235789a5545a592e23d07312ffa89 P eb6bbbb5b66f46edbffd1964a617489a [term 1 FOLLOWER]: Refusing update from remote peer 1407f59074514c53a31262544bbf537f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:03:04.610316 8062 raft_consensus.cc:1261] T 922235789a5545a592e23d07312ffa89 P b6c64a32d2274811a9476856840a6e1b [term 1 FOLLOWER]: Refusing update from remote peer 1407f59074514c53a31262544bbf537f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:03:04.610598 8435 consensus_queue.cc:1035] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:03:04.610734 8441 consensus_queue.cc:1035] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:03:04.613595 8445 mvcc.cc:205] Tried to move back new op lower bound from 7332736960960626688 to 7332736960798351360. Current Snapshot: MvccSnapshot[applied={T|T < 7332736960960626688}]
I20260924 03:03:04.614398 8446 mvcc.cc:205] Tried to move back new op lower bound from 7332736960960626688 to 7332736960798351360. Current Snapshot: MvccSnapshot[applied={T|T < 7332736960960626688 or (T in {7332736960960626688})}]
I20260924 03:03:04.616001 7918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:03:04.616103 7918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:03:04.616895 7918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22ff55de1be043bd9bee0e8586e561d8: Sending DeleteTablet for 3 replicas of tablet 922235789a5545a592e23d07312ffa89
I20260924 03:03:04.617267 8337 tablet_service.cc:1514] Processing DeleteTablet for tablet 922235789a5545a592e23d07312ffa89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:03:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40402
I20260924 03:03:04.617301 8197 tablet_service.cc:1514] Processing DeleteTablet for tablet 922235789a5545a592e23d07312ffa89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:03:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51912
I20260924 03:03:04.617316 8057 tablet_service.cc:1514] Processing DeleteTablet for tablet 922235789a5545a592e23d07312ffa89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:03:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59238
I20260924 03:03:04.617667 8456 tablet_replica.cc:330] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f: stopping tablet replica
I20260924 03:03:04.617776 8456 raft_consensus.cc:2229] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:03:04.617762 7918 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 03:03:04.617885 8456 raft_consensus.cc:2258] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 03:03:04.617911 7918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:03:04.618366 8456 ts_tablet_manager.cc:1905] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:03:04.618577 8455 tablet_replica.cc:330] T 922235789a5545a592e23d07312ffa89 P eb6bbbb5b66f46edbffd1964a617489a: stopping tablet replica
I20260924 03:03:04.618691 8455 raft_consensus.cc:2229] T 922235789a5545a592e23d07312ffa89 P eb6bbbb5b66f46edbffd1964a617489a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:03:04.618805 8455 raft_consensus.cc:2258] T 922235789a5545a592e23d07312ffa89 P eb6bbbb5b66f46edbffd1964a617489a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:03:04.619104 8057 tablet_service.cc:1467] Processing CreateTablet for tablet 29310114ba0c49cfad1d715bf404c2e5 (DEFAULT_TABLE table=test-table [id=94b3b8b228f94d0cb5c2c48412f6ea07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:03:04.619104 8337 tablet_service.cc:1467] Processing CreateTablet for tablet 29310114ba0c49cfad1d715bf404c2e5 (DEFAULT_TABLE table=test-table [id=94b3b8b228f94d0cb5c2c48412f6ea07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:03:04.619217 8057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29310114ba0c49cfad1d715bf404c2e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:03:04.619217 8337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29310114ba0c49cfad1d715bf404c2e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:03:04.619256 8455 ts_tablet_manager.cc:1905] T 922235789a5545a592e23d07312ffa89 P eb6bbbb5b66f46edbffd1964a617489a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:03:04.620674 8432 tablet_bootstrap.cc:492] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b: Bootstrap starting.
I20260924 03:03:04.620865 8197 tablet_service.cc:1467] Processing CreateTablet for tablet 29310114ba0c49cfad1d715bf404c2e5 (DEFAULT_TABLE table=test-table [id=94b3b8b228f94d0cb5c2c48412f6ea07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:03:04.620965 8197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29310114ba0c49cfad1d715bf404c2e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:03:04.621100 8432 tablet_bootstrap.cc:654] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b: Neither blocks nor log segments found. Creating new log.
I20260924 03:03:04.621200 8454 tablet_replica.cc:330] T 922235789a5545a592e23d07312ffa89 P b6c64a32d2274811a9476856840a6e1b: stopping tablet replica
I20260924 03:03:04.621313 8454 raft_consensus.cc:2229] T 922235789a5545a592e23d07312ffa89 P b6c64a32d2274811a9476856840a6e1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:03:04.621420 8454 raft_consensus.cc:2258] T 922235789a5545a592e23d07312ffa89 P b6c64a32d2274811a9476856840a6e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:03:04.621471 8431 tablet_bootstrap.cc:492] T 29310114ba0c49cfad1d715bf404c2e5 P 1407f59074514c53a31262544bbf537f: Bootstrap starting.
I20260924 03:03:04.621922 8431 tablet_bootstrap.cc:654] T 29310114ba0c49cfad1d715bf404c2e5 P 1407f59074514c53a31262544bbf537f: Neither blocks nor log segments found. Creating new log.
I20260924 03:03:04.621963 8454 ts_tablet_manager.cc:1905] T 922235789a5545a592e23d07312ffa89 P b6c64a32d2274811a9476856840a6e1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:03:04.622581 8431 tablet_bootstrap.cc:492] T 29310114ba0c49cfad1d715bf404c2e5 P 1407f59074514c53a31262544bbf537f: No bootstrap required, opened a new log
I20260924 03:03:04.622632 8431 ts_tablet_manager.cc:1397] T 29310114ba0c49cfad1d715bf404c2e5 P 1407f59074514c53a31262544bbf537f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:03:04.622774 8431 raft_consensus.cc:348] T 29310114ba0c49cfad1d715bf404c2e5 P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:04.622841 8431 raft_consensus.cc:374] T 29310114ba0c49cfad1d715bf404c2e5 P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:03:04.622860 8431 raft_consensus.cc:729] T 29310114ba0c49cfad1d715bf404c2e5 P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1407f59074514c53a31262544bbf537f, State: Initialized, Role: FOLLOWER
I20260924 03:03:04.622902 8431 consensus_queue.cc:260] T 29310114ba0c49cfad1d715bf404c2e5 P 1407f59074514c53a31262544bbf537f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:04.622974 8431 ts_tablet_manager.cc:1428] T 29310114ba0c49cfad1d715bf404c2e5 P 1407f59074514c53a31262544bbf537f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:04.623301 8437 tablet_bootstrap.cc:492] T 29310114ba0c49cfad1d715bf404c2e5 P eb6bbbb5b66f46edbffd1964a617489a: Bootstrap starting.
I20260924 03:03:04.623585 8454 ts_tablet_manager.cc:1918] T 922235789a5545a592e23d07312ffa89 P b6c64a32d2274811a9476856840a6e1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:03:04.623729 8454 log.cc:1196] T 922235789a5545a592e23d07312ffa89 P b6c64a32d2274811a9476856840a6e1b: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-0/wal/wals/922235789a5545a592e23d07312ffa89
I20260924 03:03:04.623757 8437 tablet_bootstrap.cc:654] T 29310114ba0c49cfad1d715bf404c2e5 P eb6bbbb5b66f46edbffd1964a617489a: Neither blocks nor log segments found. Creating new log.
I20260924 03:03:04.624053 8454 ts_tablet_manager.cc:1939] T 922235789a5545a592e23d07312ffa89 P b6c64a32d2274811a9476856840a6e1b: Deleting consensus metadata
I20260924 03:03:04.624395 8437 tablet_bootstrap.cc:492] T 29310114ba0c49cfad1d715bf404c2e5 P eb6bbbb5b66f46edbffd1964a617489a: No bootstrap required, opened a new log
I20260924 03:03:04.624454 8437 ts_tablet_manager.cc:1397] T 29310114ba0c49cfad1d715bf404c2e5 P eb6bbbb5b66f46edbffd1964a617489a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 03:03:04.624512 8455 ts_tablet_manager.cc:1918] T 922235789a5545a592e23d07312ffa89 P eb6bbbb5b66f46edbffd1964a617489a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:03:04.624601 8455 log.cc:1196] T 922235789a5545a592e23d07312ffa89 P eb6bbbb5b66f46edbffd1964a617489a: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-1/wal/wals/922235789a5545a592e23d07312ffa89
I20260924 03:03:04.624615 8437 raft_consensus.cc:348] T 29310114ba0c49cfad1d715bf404c2e5 P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:04.624692 8437 raft_consensus.cc:374] T 29310114ba0c49cfad1d715bf404c2e5 P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:03:04.624720 8437 raft_consensus.cc:729] T 29310114ba0c49cfad1d715bf404c2e5 P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6bbbb5b66f46edbffd1964a617489a, State: Initialized, Role: FOLLOWER
I20260924 03:03:04.624732 7907 catalog_manager.cc:4841] TS b6c64a32d2274811a9476856840a6e1b (127.0.98.1:35847): tablet 922235789a5545a592e23d07312ffa89 (table test-table [id=195a54eda1d146fabcb57305347b352a]) successfully deleted
I20260924 03:03:04.624761 8437 consensus_queue.cc:260] T 29310114ba0c49cfad1d715bf404c2e5 P eb6bbbb5b66f46edbffd1964a617489a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:04.624840 8437 ts_tablet_manager.cc:1428] T 29310114ba0c49cfad1d715bf404c2e5 P eb6bbbb5b66f46edbffd1964a617489a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 03:03:04.624883 8455 ts_tablet_manager.cc:1939] T 922235789a5545a592e23d07312ffa89 P eb6bbbb5b66f46edbffd1964a617489a: Deleting consensus metadata
I20260924 03:03:04.625382 7907 catalog_manager.cc:4841] TS eb6bbbb5b66f46edbffd1964a617489a (127.0.98.2:39853): tablet 922235789a5545a592e23d07312ffa89 (table test-table [id=195a54eda1d146fabcb57305347b352a]) successfully deleted
I20260924 03:03:04.626740 8456 ts_tablet_manager.cc:1918] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:03:04.626819 8456 log.cc:1196] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-2/wal/wals/922235789a5545a592e23d07312ffa89
I20260924 03:03:04.626824 8432 tablet_bootstrap.cc:492] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b: No bootstrap required, opened a new log
I20260924 03:03:04.626869 8432 ts_tablet_manager.cc:1397] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260924 03:03:04.627009 8432 raft_consensus.cc:348] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:04.627086 8432 raft_consensus.cc:374] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:03:04.627107 8432 raft_consensus.cc:729] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6c64a32d2274811a9476856840a6e1b, State: Initialized, Role: FOLLOWER
I20260924 03:03:04.627140 8456 ts_tablet_manager.cc:1939] T 922235789a5545a592e23d07312ffa89 P 1407f59074514c53a31262544bbf537f: Deleting consensus metadata
I20260924 03:03:04.627152 8432 consensus_queue.cc:260] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:04.627461 8432 ts_tablet_manager.cc:1428] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:03:04.627513 7907 catalog_manager.cc:4841] TS 1407f59074514c53a31262544bbf537f (127.0.98.3:39147): tablet 922235789a5545a592e23d07312ffa89 (table test-table [id=195a54eda1d146fabcb57305347b352a]) successfully deleted
W20260924 03:03:04.630873 8404 tablet.cc:2369] T 29310114ba0c49cfad1d715bf404c2e5 P 1407f59074514c53a31262544bbf537f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:03:04.746786 8436 raft_consensus.cc:482] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:03:04.746954 8436 raft_consensus.cc:504] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:04.747246 8436 leader_election.cc:288] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6bbbb5b66f46edbffd1964a617489a (127.0.98.2:39853), 1407f59074514c53a31262544bbf537f (127.0.98.3:39147)
I20260924 03:03:04.750540 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.747636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44551 (local address 127.0.98.2:39853)
0924 03:03:04.747699 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:04.747701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:04.747802 (+ 101us) server_negotiation.cc:407] Connection header received
0924 03:03:04.747884 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:04.747885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:04.747895 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:04.747955 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:03:04.748184 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.748760 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.749813 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.750171 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.750252 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.750263 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:04.750423 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 03:03:04.750466 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 03:03:04.750560 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.747822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39749 (local address 127.0.98.3:39147)
0924 03:03:04.747877 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:04.747880 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:04.747909 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:03:04.748052 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:04.748053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:04.748061 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:04.748139 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:03:04.748290 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.748752 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.749809 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.750179 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.750257 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.750267 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:04.750454 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 03:03:04.750487 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 03:03:04.750583 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.747512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39853 (local address 127.0.98.1:44551)
0924 03:03:04.747697 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:04.747727 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:04.747808 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:04.748027 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:04.748028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:04.748030 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:03:04.748099 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.748100 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.748846 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.748847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.749713 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.749715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.750251 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.750252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.750340 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.750343 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:03:04.750415 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 03:03:04.750452 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":25,"thread_start_us":106,"threads_started":1}
I20260924 03:03:04.750589 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.747743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39147 (local address 127.0.98.1:39749)
0924 03:03:04.747854 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:04.747864 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:04.747925 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:04.748142 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:04.748143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:04.748144 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:03:04.748214 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.748215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.748835 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.748836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.749715 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.749717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.750279 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.750281 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.750369 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.750373 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:03:04.750447 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 03:03:04.750491 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":61,"thread_start_us":59,"threads_started":1}
I20260924 03:03:04.750847 8217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29310114ba0c49cfad1d715bf404c2e5" candidate_uuid: "b6c64a32d2274811a9476856840a6e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6bbbb5b66f46edbffd1964a617489a" is_pre_election: true
I20260924 03:03:04.750948 8217 raft_consensus.cc:2454] T 29310114ba0c49cfad1d715bf404c2e5 P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6c64a32d2274811a9476856840a6e1b in term 0.
I20260924 03:03:04.751156 8011 leader_election.cc:302] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6c64a32d2274811a9476856840a6e1b, eb6bbbb5b66f46edbffd1964a617489a; no voters:
I20260924 03:03:04.751199 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29310114ba0c49cfad1d715bf404c2e5" candidate_uuid: "b6c64a32d2274811a9476856840a6e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1407f59074514c53a31262544bbf537f" is_pre_election: true
I20260924 03:03:04.751299 8436 raft_consensus.cc:2781] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:03:04.751343 8436 raft_consensus.cc:482] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:03:04.751348 8357 raft_consensus.cc:2454] T 29310114ba0c49cfad1d715bf404c2e5 P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6c64a32d2274811a9476856840a6e1b in term 0.
I20260924 03:03:04.751370 8436 raft_consensus.cc:3037] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:03:04.752137 8436 raft_consensus.cc:504] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:04.752278 8436 leader_election.cc:288] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b [CANDIDATE]: Term 1 election: Requested vote from peers eb6bbbb5b66f46edbffd1964a617489a (127.0.98.2:39853), 1407f59074514c53a31262544bbf537f (127.0.98.3:39147)
I20260924 03:03:04.752496 8217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29310114ba0c49cfad1d715bf404c2e5" candidate_uuid: "b6c64a32d2274811a9476856840a6e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6bbbb5b66f46edbffd1964a617489a"
I20260924 03:03:04.752588 8217 raft_consensus.cc:3037] T 29310114ba0c49cfad1d715bf404c2e5 P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:03:04.752566 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29310114ba0c49cfad1d715bf404c2e5" candidate_uuid: "b6c64a32d2274811a9476856840a6e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1407f59074514c53a31262544bbf537f"
I20260924 03:03:04.752650 8357 raft_consensus.cc:3037] T 29310114ba0c49cfad1d715bf404c2e5 P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:03:04.753376 8217 raft_consensus.cc:2454] T 29310114ba0c49cfad1d715bf404c2e5 P eb6bbbb5b66f46edbffd1964a617489a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6c64a32d2274811a9476856840a6e1b in term 1.
I20260924 03:03:04.753518 8357 raft_consensus.cc:2454] T 29310114ba0c49cfad1d715bf404c2e5 P 1407f59074514c53a31262544bbf537f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6c64a32d2274811a9476856840a6e1b in term 1.
I20260924 03:03:04.753636 8011 leader_election.cc:302] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6c64a32d2274811a9476856840a6e1b, eb6bbbb5b66f46edbffd1964a617489a; no voters:
I20260924 03:03:04.753739 8436 raft_consensus.cc:2781] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:03:04.753916 8436 raft_consensus.cc:686] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b [term 1 LEADER]: Becoming Leader. State: Replica: b6c64a32d2274811a9476856840a6e1b, State: Running, Role: LEADER
I20260924 03:03:04.754034 8436 consensus_queue.cc:237] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:04.754925 7918 catalog_manager.cc:5495] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b reported cstate change: term changed from 0 to 1, leader changed from <none> to b6c64a32d2274811a9476856840a6e1b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b6c64a32d2274811a9476856840a6e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } health_report { overall_health: UNKNOWN } } }
Sep 24 03:03:04 dist-test-slave-wgsb krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218983, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:03:04.795042 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.788326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35620 (local address 127.0.98.62:33095)
0924 03:03:04.788412 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:04.788415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:04.788533 (+ 118us) server_negotiation.cc:407] Connection header received
0924 03:03:04.788621 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:04.788622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:04.788633 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:04.788702 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:04.789148 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.789663 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.790354 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.790533 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.790621 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.792750 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:04.792751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:04.792751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:03:04.792758 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:04.793967 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:04.794364 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:04.794365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:04.794365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:04.794410 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:04.794614 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:04.794615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:04.794615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:04.794714 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:04.794765 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:04.794911 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 03:03:04.794975 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 03:03:04.795205 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:04.788251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33095 (local address 127.0.0.1:35620)
0924 03:03:04.788453 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:04.788471 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:04.788561 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:04.788705 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:04.788706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:04.789012 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:04.789056 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.789059 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.789745 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.789746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.790288 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:04.790290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:04.790634 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:04.790635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:04.790691 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:04.791224 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:04.791229 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:04.792664 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:03:04.794073 (+ 1409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:03:04.794079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:03:04.794081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:03:04.794296 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:03:04.794469 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:03:04.794470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:03:04.794470 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:03:04.794537 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:03:04.794773 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:03:04.794779 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:03:04.794825 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 03:03:04.794881 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 03:03:04.795035 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":28,"thread_start_us":163,"threads_started":1}
I20260924 03:03:04.797222 7918 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35620:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 03:03:04.797402 7918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:03:04.798885 8337 tablet_service.cc:1467] Processing CreateTablet for tablet d82bdf343dc2463395f8fcef272c6fcf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=32a9f3b771bd4db0ae55bcec37538f80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:03:04.798864 8057 tablet_service.cc:1467] Processing CreateTablet for tablet d82bdf343dc2463395f8fcef272c6fcf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=32a9f3b771bd4db0ae55bcec37538f80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:03:04.799031 8337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d82bdf343dc2463395f8fcef272c6fcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:03:04.799031 8057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d82bdf343dc2463395f8fcef272c6fcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:03:04.800539 8431 tablet_bootstrap.cc:492] T d82bdf343dc2463395f8fcef272c6fcf P 1407f59074514c53a31262544bbf537f: Bootstrap starting.
I20260924 03:03:04.800539 8432 tablet_bootstrap.cc:492] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b: Bootstrap starting.
I20260924 03:03:04.800634 8197 tablet_service.cc:1467] Processing CreateTablet for tablet d82bdf343dc2463395f8fcef272c6fcf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=32a9f3b771bd4db0ae55bcec37538f80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:03:04.800762 8197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d82bdf343dc2463395f8fcef272c6fcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:03:04.800925 8431 tablet_bootstrap.cc:654] T d82bdf343dc2463395f8fcef272c6fcf P 1407f59074514c53a31262544bbf537f: Neither blocks nor log segments found. Creating new log.
I20260924 03:03:04.800946 8432 tablet_bootstrap.cc:654] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b: Neither blocks nor log segments found. Creating new log.
I20260924 03:03:04.801496 8431 tablet_bootstrap.cc:492] T d82bdf343dc2463395f8fcef272c6fcf P 1407f59074514c53a31262544bbf537f: No bootstrap required, opened a new log
I20260924 03:03:04.801563 8431 ts_tablet_manager.cc:1397] T d82bdf343dc2463395f8fcef272c6fcf P 1407f59074514c53a31262544bbf537f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:03:04.801681 8431 raft_consensus.cc:348] T d82bdf343dc2463395f8fcef272c6fcf P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:04.801756 8431 raft_consensus.cc:374] T d82bdf343dc2463395f8fcef272c6fcf P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:03:04.801788 8431 raft_consensus.cc:729] T d82bdf343dc2463395f8fcef272c6fcf P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1407f59074514c53a31262544bbf537f, State: Initialized, Role: FOLLOWER
I20260924 03:03:04.801838 8431 consensus_queue.cc:260] T d82bdf343dc2463395f8fcef272c6fcf P 1407f59074514c53a31262544bbf537f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:04.801936 8431 ts_tablet_manager.cc:1428] T d82bdf343dc2463395f8fcef272c6fcf P 1407f59074514c53a31262544bbf537f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:04.801982 8449 tablet_replica.cc:439] T d82bdf343dc2463395f8fcef272c6fcf P 1407f59074514c53a31262544bbf537f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } } }
I20260924 03:03:04.802078 8449 tablet_replica.cc:442] T d82bdf343dc2463395f8fcef272c6fcf P 1407f59074514c53a31262544bbf537f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:03:04.802214 8437 tablet_bootstrap.cc:492] T d82bdf343dc2463395f8fcef272c6fcf P eb6bbbb5b66f46edbffd1964a617489a: Bootstrap starting.
I20260924 03:03:04.802450 8432 tablet_bootstrap.cc:492] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b: No bootstrap required, opened a new log
I20260924 03:03:04.802505 8432 ts_tablet_manager.cc:1397] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:03:04.802608 8437 tablet_bootstrap.cc:654] T d82bdf343dc2463395f8fcef272c6fcf P eb6bbbb5b66f46edbffd1964a617489a: Neither blocks nor log segments found. Creating new log.
I20260924 03:03:04.802663 8432 raft_consensus.cc:348] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:04.802742 8432 raft_consensus.cc:374] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:03:04.802755 8432 raft_consensus.cc:729] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6c64a32d2274811a9476856840a6e1b, State: Initialized, Role: FOLLOWER
I20260924 03:03:04.802806 8432 consensus_queue.cc:260] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:04.802896 8432 ts_tablet_manager.cc:1428] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:04.802932 8436 tablet_replica.cc:439] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } } }
I20260924 03:03:04.802999 8436 tablet_replica.cc:442] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:03:04.803341 8437 tablet_bootstrap.cc:492] T d82bdf343dc2463395f8fcef272c6fcf P eb6bbbb5b66f46edbffd1964a617489a: No bootstrap required, opened a new log
I20260924 03:03:04.803399 8437 ts_tablet_manager.cc:1397] T d82bdf343dc2463395f8fcef272c6fcf P eb6bbbb5b66f46edbffd1964a617489a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 03:03:04.803544 8437 raft_consensus.cc:348] T d82bdf343dc2463395f8fcef272c6fcf P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:04.803637 8437 raft_consensus.cc:374] T d82bdf343dc2463395f8fcef272c6fcf P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:03:04.803665 8437 raft_consensus.cc:729] T d82bdf343dc2463395f8fcef272c6fcf P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6bbbb5b66f46edbffd1964a617489a, State: Initialized, Role: FOLLOWER
I20260924 03:03:04.803717 8437 consensus_queue.cc:260] T d82bdf343dc2463395f8fcef272c6fcf P eb6bbbb5b66f46edbffd1964a617489a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:04.803814 8437 ts_tablet_manager.cc:1428] T d82bdf343dc2463395f8fcef272c6fcf P eb6bbbb5b66f46edbffd1964a617489a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:04.803876 8439 tablet_replica.cc:439] T d82bdf343dc2463395f8fcef272c6fcf P eb6bbbb5b66f46edbffd1964a617489a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } } }
I20260924 03:03:04.803978 8439 tablet_replica.cc:442] T d82bdf343dc2463395f8fcef272c6fcf P eb6bbbb5b66f46edbffd1964a617489a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:03:05.073462 8436 raft_consensus.cc:482] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:03:05.073563 8436 raft_consensus.cc:504] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:05.073721 8436 leader_election.cc:288] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6bbbb5b66f46edbffd1964a617489a (127.0.98.2:39853), 1407f59074514c53a31262544bbf537f (127.0.98.3:39147)
I20260924 03:03:05.073975 8217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d82bdf343dc2463395f8fcef272c6fcf" candidate_uuid: "b6c64a32d2274811a9476856840a6e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6bbbb5b66f46edbffd1964a617489a" is_pre_election: true
I20260924 03:03:05.074074 8217 raft_consensus.cc:2454] T d82bdf343dc2463395f8fcef272c6fcf P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6c64a32d2274811a9476856840a6e1b in term 0.
I20260924 03:03:05.074036 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d82bdf343dc2463395f8fcef272c6fcf" candidate_uuid: "b6c64a32d2274811a9476856840a6e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1407f59074514c53a31262544bbf537f" is_pre_election: true
I20260924 03:03:05.074160 8357 raft_consensus.cc:2454] T d82bdf343dc2463395f8fcef272c6fcf P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6c64a32d2274811a9476856840a6e1b in term 0.
I20260924 03:03:05.074223 8011 leader_election.cc:302] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6c64a32d2274811a9476856840a6e1b, eb6bbbb5b66f46edbffd1964a617489a; no voters:
I20260924 03:03:05.074313 8436 raft_consensus.cc:2781] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:03:05.074354 8436 raft_consensus.cc:482] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:03:05.074374 8436 raft_consensus.cc:3037] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:03:05.075085 8436 raft_consensus.cc:504] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:05.075217 8436 leader_election.cc:288] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b [CANDIDATE]: Term 1 election: Requested vote from peers eb6bbbb5b66f46edbffd1964a617489a (127.0.98.2:39853), 1407f59074514c53a31262544bbf537f (127.0.98.3:39147)
I20260924 03:03:05.075397 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d82bdf343dc2463395f8fcef272c6fcf" candidate_uuid: "b6c64a32d2274811a9476856840a6e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1407f59074514c53a31262544bbf537f"
I20260924 03:03:05.075397 8217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d82bdf343dc2463395f8fcef272c6fcf" candidate_uuid: "b6c64a32d2274811a9476856840a6e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6bbbb5b66f46edbffd1964a617489a"
I20260924 03:03:05.075475 8357 raft_consensus.cc:3037] T d82bdf343dc2463395f8fcef272c6fcf P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:03:05.075469 8217 raft_consensus.cc:3037] T d82bdf343dc2463395f8fcef272c6fcf P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:03:05.076220 8357 raft_consensus.cc:2454] T d82bdf343dc2463395f8fcef272c6fcf P 1407f59074514c53a31262544bbf537f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6c64a32d2274811a9476856840a6e1b in term 1.
I20260924 03:03:05.076220 8217 raft_consensus.cc:2454] T d82bdf343dc2463395f8fcef272c6fcf P eb6bbbb5b66f46edbffd1964a617489a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6c64a32d2274811a9476856840a6e1b in term 1.
I20260924 03:03:05.076385 8011 leader_election.cc:302] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6c64a32d2274811a9476856840a6e1b, eb6bbbb5b66f46edbffd1964a617489a; no voters:
I20260924 03:03:05.076529 8436 raft_consensus.cc:2781] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:03:05.076602 8436 raft_consensus.cc:686] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b [term 1 LEADER]: Becoming Leader. State: Replica: b6c64a32d2274811a9476856840a6e1b, State: Running, Role: LEADER
I20260924 03:03:05.076654 8436 consensus_queue.cc:237] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:05.076936 8462 tablet_replica.cc:439] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b: TxnStatusTablet state changed. Reason: New leader b6c64a32d2274811a9476856840a6e1b. Latest consensus state: current_term: 1 leader_uuid: "b6c64a32d2274811a9476856840a6e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } } }
I20260924 03:03:05.077069 8462 tablet_replica.cc:442] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:03:05.077248 8474 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 03:03:05.077279 7918 catalog_manager.cc:5495] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b reported cstate change: term changed from 0 to 1, leader changed from <none> to b6c64a32d2274811a9476856840a6e1b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b6c64a32d2274811a9476856840a6e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } health_report { overall_health: UNKNOWN } } }
I20260924 03:03:05.077378 8474 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 03:03:05.087580 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:05.084917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35847 (local address 127.0.0.1:59240)
0924 03:03:05.084999 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:05.085009 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:05.085044 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:05.085306 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:05.085307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:05.085308 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:03:05.085358 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:05.085360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:05.086035 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:05.086036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:05.086620 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:05.086622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:05.087033 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:05.087034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:05.087108 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:05.087128 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:03:05.087407 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:03:05.087416 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 03:03:05.087461 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 03:03:05.087507 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 03:03:05.087635 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:05.084975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59240 (local address 127.0.98.1:35847)
0924 03:03:05.085147 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:05.085149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:05.085156 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:03:05.085190 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:05.085191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:05.085209 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:05.085302 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:03:05.085410 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:05.085950 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:05.086716 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:05.086957 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:05.087034 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:05.087230 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:03:05.087337 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:03:05.087421 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:05.087474 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 03:03:05.087497 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":82,"threads_started":1}
I20260924 03:03:05.089370 8217 raft_consensus.cc:1261] T d82bdf343dc2463395f8fcef272c6fcf P eb6bbbb5b66f46edbffd1964a617489a [term 1 FOLLOWER]: Refusing update from remote peer b6c64a32d2274811a9476856840a6e1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:03:05.089462 8357 raft_consensus.cc:1261] T d82bdf343dc2463395f8fcef272c6fcf P 1407f59074514c53a31262544bbf537f [term 1 FOLLOWER]: Refusing update from remote peer b6c64a32d2274811a9476856840a6e1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:03:05.089816 8462 consensus_queue.cc:1035] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:03:05.089991 8462 consensus_queue.cc:1035] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:03:05.090617 8439 tablet_replica.cc:439] T d82bdf343dc2463395f8fcef272c6fcf P eb6bbbb5b66f46edbffd1964a617489a: TxnStatusTablet state changed. Reason: New leader b6c64a32d2274811a9476856840a6e1b. Latest consensus state: current_term: 1 leader_uuid: "b6c64a32d2274811a9476856840a6e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } } }
I20260924 03:03:05.090701 8439 tablet_replica.cc:442] T d82bdf343dc2463395f8fcef272c6fcf P eb6bbbb5b66f46edbffd1964a617489a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:03:05.091173 8462 tablet_replica.cc:439] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b6c64a32d2274811a9476856840a6e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } } }
I20260924 03:03:05.091260 8462 tablet_replica.cc:442] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:03:05.092092 8449 tablet_replica.cc:439] T d82bdf343dc2463395f8fcef272c6fcf P 1407f59074514c53a31262544bbf537f: TxnStatusTablet state changed. Reason: New leader b6c64a32d2274811a9476856840a6e1b. Latest consensus state: current_term: 1 leader_uuid: "b6c64a32d2274811a9476856840a6e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } } }
I20260924 03:03:05.092183 8449 tablet_replica.cc:442] T d82bdf343dc2463395f8fcef272c6fcf P 1407f59074514c53a31262544bbf537f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:03:05.092259 8439 tablet_replica.cc:439] T d82bdf343dc2463395f8fcef272c6fcf P eb6bbbb5b66f46edbffd1964a617489a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b6c64a32d2274811a9476856840a6e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } } }
I20260924 03:03:05.092339 8439 tablet_replica.cc:442] T d82bdf343dc2463395f8fcef272c6fcf P eb6bbbb5b66f46edbffd1964a617489a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:03:05.093065 8436 tablet_replica.cc:439] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b6c64a32d2274811a9476856840a6e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } } }
I20260924 03:03:05.093158 8436 tablet_replica.cc:442] T d82bdf343dc2463395f8fcef272c6fcf P b6c64a32d2274811a9476856840a6e1b: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:03:05.093684 8449 tablet_replica.cc:439] T d82bdf343dc2463395f8fcef272c6fcf P 1407f59074514c53a31262544bbf537f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b6c64a32d2274811a9476856840a6e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } } }
I20260924 03:03:05.093763 8449 tablet_replica.cc:442] T d82bdf343dc2463395f8fcef272c6fcf P 1407f59074514c53a31262544bbf537f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:03:05.095649 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:05.093133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59248 (local address 127.0.98.1:35847)
0924 03:03:05.093232 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:05.093234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:05.093240 (+ 6us) server_negotiation.cc:407] Connection header received
0924 03:03:05.093262 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:05.093263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:05.093287 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:05.093337 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:03:05.093503 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:05.094228 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:05.094943 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:05.095152 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:05.095224 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:05.095395 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:03:05.095425 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:03:05.095478 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:05.095539 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 03:03:05.095559 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 03:03:05.100510 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:05.097840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59117 (local address 127.0.98.1:35847)
0924 03:03:05.097876 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:05.097878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:05.098011 (+ 133us) server_negotiation.cc:407] Connection header received
0924 03:03:05.098102 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:05.098103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:05.098111 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:05.098173 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:03:05.098297 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:05.098710 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:05.099856 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:05.100026 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:05.100111 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:05.100295 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:03:05.100317 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:03:05.100349 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:05.100423 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 03:03:05.100439 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":20}
I20260924 03:03:05.100605 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:05.097762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35847 (local address 127.0.98.1:59117)
0924 03:03:05.097926 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:05.097937 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:05.098011 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:05.098177 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:05.098178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:05.098180 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:03:05.098243 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:05.098244 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:05.098776 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:05.098777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:05.099781 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:05.099783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:05.100142 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:05.100143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:05.100205 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:05.100219 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:03:05.100368 (+ 149us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:03:05.100370 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:03:05.100421 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 03:03:05.100463 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
I20260924 03:03:05.102614 8357 raft_consensus.cc:1261] T 29310114ba0c49cfad1d715bf404c2e5 P 1407f59074514c53a31262544bbf537f [term 1 FOLLOWER]: Refusing update from remote peer b6c64a32d2274811a9476856840a6e1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:03:05.102632 8217 raft_consensus.cc:1261] T 29310114ba0c49cfad1d715bf404c2e5 P eb6bbbb5b66f46edbffd1964a617489a [term 1 FOLLOWER]: Refusing update from remote peer b6c64a32d2274811a9476856840a6e1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:03:05.102814 8480 consensus_queue.cc:1035] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:03:05.102929 8436 consensus_queue.cc:1035] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:03:05.117349 7920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:03:05.117467 7920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:03:05.118088 7920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22ff55de1be043bd9bee0e8586e561d8: Sending DeleteTablet for 3 replicas of tablet 29310114ba0c49cfad1d715bf404c2e5
I20260924 03:03:05.118314 8057 tablet_service.cc:1514] Processing DeleteTablet for tablet 29310114ba0c49cfad1d715bf404c2e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:03:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59238
I20260924 03:03:05.118377 8337 tablet_service.cc:1514] Processing DeleteTablet for tablet 29310114ba0c49cfad1d715bf404c2e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:03:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40402
I20260924 03:03:05.118412 8454 tablet_replica.cc:330] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b: stopping tablet replica
I20260924 03:03:05.118455 8454 raft_consensus.cc:2229] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:03:05.118486 8456 tablet_replica.cc:330] T 29310114ba0c49cfad1d715bf404c2e5 P 1407f59074514c53a31262544bbf537f: stopping tablet replica
I20260924 03:03:05.118533 8456 raft_consensus.cc:2229] T 29310114ba0c49cfad1d715bf404c2e5 P 1407f59074514c53a31262544bbf537f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:03:05.118539 8454 raft_consensus.cc:2258] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:03:05.118570 8456 raft_consensus.cc:2258] T 29310114ba0c49cfad1d715bf404c2e5 P 1407f59074514c53a31262544bbf537f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:03:05.118891 8454 ts_tablet_manager.cc:1905] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:03:05.119738 8197 tablet_service.cc:1514] Processing DeleteTablet for tablet 29310114ba0c49cfad1d715bf404c2e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:03:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51912
I20260924 03:03:05.120316 8455 tablet_replica.cc:330] T 29310114ba0c49cfad1d715bf404c2e5 P eb6bbbb5b66f46edbffd1964a617489a: stopping tablet replica
I20260924 03:03:05.120407 8455 raft_consensus.cc:2229] T 29310114ba0c49cfad1d715bf404c2e5 P eb6bbbb5b66f46edbffd1964a617489a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:03:05.120466 8455 raft_consensus.cc:2258] T 29310114ba0c49cfad1d715bf404c2e5 P eb6bbbb5b66f46edbffd1964a617489a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:03:05.120620 8454 ts_tablet_manager.cc:1918] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:03:05.120675 8454 log.cc:1196] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-0/wal/wals/29310114ba0c49cfad1d715bf404c2e5
I20260924 03:03:05.120898 8455 ts_tablet_manager.cc:1905] T 29310114ba0c49cfad1d715bf404c2e5 P eb6bbbb5b66f46edbffd1964a617489a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:03:05.120915 8454 ts_tablet_manager.cc:1939] T 29310114ba0c49cfad1d715bf404c2e5 P b6c64a32d2274811a9476856840a6e1b: Deleting consensus metadata
I20260924 03:03:05.121447 8456 ts_tablet_manager.cc:1905] T 29310114ba0c49cfad1d715bf404c2e5 P 1407f59074514c53a31262544bbf537f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:03:05.122421 8455 ts_tablet_manager.cc:1918] T 29310114ba0c49cfad1d715bf404c2e5 P eb6bbbb5b66f46edbffd1964a617489a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:03:05.122474 8455 log.cc:1196] T 29310114ba0c49cfad1d715bf404c2e5 P eb6bbbb5b66f46edbffd1964a617489a: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-1/wal/wals/29310114ba0c49cfad1d715bf404c2e5
I20260924 03:03:05.122685 8455 ts_tablet_manager.cc:1939] T 29310114ba0c49cfad1d715bf404c2e5 P eb6bbbb5b66f46edbffd1964a617489a: Deleting consensus metadata
I20260924 03:03:05.122905 7907 catalog_manager.cc:4841] TS b6c64a32d2274811a9476856840a6e1b (127.0.98.1:35847): tablet 29310114ba0c49cfad1d715bf404c2e5 (table test-table [id=94b3b8b228f94d0cb5c2c48412f6ea07]) successfully deleted
I20260924 03:03:05.123100 8456 ts_tablet_manager.cc:1918] T 29310114ba0c49cfad1d715bf404c2e5 P 1407f59074514c53a31262544bbf537f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:03:05.123160 8456 log.cc:1196] T 29310114ba0c49cfad1d715bf404c2e5 P 1407f59074514c53a31262544bbf537f: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-2/wal/wals/29310114ba0c49cfad1d715bf404c2e5
I20260924 03:03:05.123345 8456 ts_tablet_manager.cc:1939] T 29310114ba0c49cfad1d715bf404c2e5 P 1407f59074514c53a31262544bbf537f: Deleting consensus metadata
I20260924 03:03:05.123332 7907 catalog_manager.cc:4841] TS eb6bbbb5b66f46edbffd1964a617489a (127.0.98.2:39853): tablet 29310114ba0c49cfad1d715bf404c2e5 (table test-table [id=94b3b8b228f94d0cb5c2c48412f6ea07]) successfully deleted
I20260924 03:03:05.123925 7907 catalog_manager.cc:4841] TS 1407f59074514c53a31262544bbf537f (127.0.98.3:39147): tablet 29310114ba0c49cfad1d715bf404c2e5 (table test-table [id=94b3b8b228f94d0cb5c2c48412f6ea07]) successfully deleted
Sep 24 03:03:05 dist-test-slave-wgsb krb5kdc[7875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 03:03:05 dist-test-slave-wgsb krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:05 dist-test-slave-wgsb krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 03:03:05.144352 7987 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:05.138612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35632 (local address 127.0.98.62:33095)
0924 03:03:05.138664 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:05.138667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:05.138766 (+ 99us) server_negotiation.cc:407] Connection header received
0924 03:03:05.138887 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:05.138888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:05.138900 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:05.138976 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:05.139721 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:05.140260 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:05.141039 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:05.141236 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:05.141323 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:05.144279 (+ 2956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35632: BlockingRecv error: recv got EOF from 127.0.0.1:35632 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 24 03:03:05 dist-test-slave-wgsb krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 03:03:05.152249 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:05.145235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35640 (local address 127.0.98.62:33095)
0924 03:03:05.145284 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:05.145286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:05.145293 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:03:05.145368 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:05.145369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:05.145378 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:05.145438 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:05.145914 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:05.146377 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:05.147125 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:05.147327 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:05.147411 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:05.149594 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:05.149596 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:05.149596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:03:05.149604 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:05.151005 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:05.151362 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:05.151363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:05.151364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:05.151421 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:05.151668 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:05.151669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:05.151670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:05.151786 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:05.151894 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:05.152056 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 03:03:05.152136 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 03:03:05.153815 7920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:03:05.153966 7920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:03:05.155462 8057 tablet_service.cc:1467] Processing CreateTablet for tablet 3bda2b4926ae4d34addcfb0045d7cc63 (DEFAULT_TABLE table=test-table [id=33f0fde9437f41d38ceca69686fcbc52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:03:05.155505 8337 tablet_service.cc:1467] Processing CreateTablet for tablet 3bda2b4926ae4d34addcfb0045d7cc63 (DEFAULT_TABLE table=test-table [id=33f0fde9437f41d38ceca69686fcbc52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:03:05.155644 8057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bda2b4926ae4d34addcfb0045d7cc63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:03:05.155750 8337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bda2b4926ae4d34addcfb0045d7cc63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:03:05.156916 8432 tablet_bootstrap.cc:492] T 3bda2b4926ae4d34addcfb0045d7cc63 P b6c64a32d2274811a9476856840a6e1b: Bootstrap starting.
I20260924 03:03:05.157008 8197 tablet_service.cc:1467] Processing CreateTablet for tablet 3bda2b4926ae4d34addcfb0045d7cc63 (DEFAULT_TABLE table=test-table [id=33f0fde9437f41d38ceca69686fcbc52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:03:05.157133 8197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bda2b4926ae4d34addcfb0045d7cc63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:03:05.157208 8431 tablet_bootstrap.cc:492] T 3bda2b4926ae4d34addcfb0045d7cc63 P 1407f59074514c53a31262544bbf537f: Bootstrap starting.
I20260924 03:03:05.157231 8432 tablet_bootstrap.cc:654] T 3bda2b4926ae4d34addcfb0045d7cc63 P b6c64a32d2274811a9476856840a6e1b: Neither blocks nor log segments found. Creating new log.
I20260924 03:03:05.157583 8431 tablet_bootstrap.cc:654] T 3bda2b4926ae4d34addcfb0045d7cc63 P 1407f59074514c53a31262544bbf537f: Neither blocks nor log segments found. Creating new log.
I20260924 03:03:05.158120 8431 tablet_bootstrap.cc:492] T 3bda2b4926ae4d34addcfb0045d7cc63 P 1407f59074514c53a31262544bbf537f: No bootstrap required, opened a new log
I20260924 03:03:05.158166 8431 ts_tablet_manager.cc:1397] T 3bda2b4926ae4d34addcfb0045d7cc63 P 1407f59074514c53a31262544bbf537f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:03:05.158259 8432 tablet_bootstrap.cc:492] T 3bda2b4926ae4d34addcfb0045d7cc63 P b6c64a32d2274811a9476856840a6e1b: No bootstrap required, opened a new log
I20260924 03:03:05.158303 8432 ts_tablet_manager.cc:1397] T 3bda2b4926ae4d34addcfb0045d7cc63 P b6c64a32d2274811a9476856840a6e1b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:03:05.158301 8431 raft_consensus.cc:348] T 3bda2b4926ae4d34addcfb0045d7cc63 P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:05.158373 8431 raft_consensus.cc:374] T 3bda2b4926ae4d34addcfb0045d7cc63 P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:03:05.158398 8431 raft_consensus.cc:729] T 3bda2b4926ae4d34addcfb0045d7cc63 P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1407f59074514c53a31262544bbf537f, State: Initialized, Role: FOLLOWER
I20260924 03:03:05.158408 8432 raft_consensus.cc:348] T 3bda2b4926ae4d34addcfb0045d7cc63 P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:05.158473 8432 raft_consensus.cc:374] T 3bda2b4926ae4d34addcfb0045d7cc63 P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:03:05.158455 8431 consensus_queue.cc:260] T 3bda2b4926ae4d34addcfb0045d7cc63 P 1407f59074514c53a31262544bbf537f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:05.158496 8432 raft_consensus.cc:729] T 3bda2b4926ae4d34addcfb0045d7cc63 P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6c64a32d2274811a9476856840a6e1b, State: Initialized, Role: FOLLOWER
I20260924 03:03:05.158540 8432 consensus_queue.cc:260] T 3bda2b4926ae4d34addcfb0045d7cc63 P b6c64a32d2274811a9476856840a6e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:05.158556 8431 ts_tablet_manager.cc:1428] T 3bda2b4926ae4d34addcfb0045d7cc63 P 1407f59074514c53a31262544bbf537f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:05.158630 8432 ts_tablet_manager.cc:1428] T 3bda2b4926ae4d34addcfb0045d7cc63 P b6c64a32d2274811a9476856840a6e1b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:05.158636 8437 tablet_bootstrap.cc:492] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a: Bootstrap starting.
I20260924 03:03:05.159055 8437 tablet_bootstrap.cc:654] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a: Neither blocks nor log segments found. Creating new log.
I20260924 03:03:05.159736 8437 tablet_bootstrap.cc:492] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a: No bootstrap required, opened a new log
I20260924 03:03:05.159788 8437 ts_tablet_manager.cc:1397] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 03:03:05.160027 8437 raft_consensus.cc:348] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:05.160151 8437 raft_consensus.cc:374] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:03:05.160192 8437 raft_consensus.cc:729] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6bbbb5b66f46edbffd1964a617489a, State: Initialized, Role: FOLLOWER
I20260924 03:03:05.160250 8437 consensus_queue.cc:260] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:05.160349 8437 ts_tablet_manager.cc:1428] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 03:03:05.357092 8439 raft_consensus.cc:482] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:03:05.357203 8439 raft_consensus.cc:504] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:05.357553 8439 leader_election.cc:288] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6c64a32d2274811a9476856840a6e1b (127.0.98.1:35847), 1407f59074514c53a31262544bbf537f (127.0.98.3:39147)
I20260924 03:03:05.360826 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:05.357799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35847 (local address 127.0.98.2:43555)
0924 03:03:05.357967 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:05.357979 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:05.358084 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:05.358272 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:05.358273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:05.358275 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:03:05.358343 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:05.358346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:05.359017 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:05.359018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:05.360000 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:05.360002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:05.360584 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:05.360585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:05.360649 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:05.360654 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:03:05.360707 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 03:03:05.360748 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":30,"thread_start_us":56,"threads_started":1}
I20260924 03:03:05.360841 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:05.357884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43555 (local address 127.0.98.1:35847)
0924 03:03:05.357945 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:05.357948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:05.358053 (+ 105us) server_negotiation.cc:407] Connection header received
0924 03:03:05.358175 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:05.358177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:05.358187 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:05.358250 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:03:05.358414 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:05.358941 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:05.360087 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:05.360496 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:05.360549 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:05.360558 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:05.360726 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 03:03:05.360762 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 03:03:05.360998 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:05.358116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35395 (local address 127.0.98.3:39147)
0924 03:03:05.358247 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:05.358249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:05.358258 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:03:05.358281 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:05.358282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:05.358292 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:05.358373 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:03:05.358548 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:05.359194 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:05.360220 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:05.360643 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:05.360715 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:05.360727 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:05.360893 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 03:03:05.360937 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260924 03:03:05.361224 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:05.357968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39147 (local address 127.0.98.2:35395)
0924 03:03:05.358080 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:05.358091 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:05.358169 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:05.358405 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:05.358407 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:05.358409 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:03:05.358470 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:05.358471 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:05.359283 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:05.359284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:05.360139 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:05.360142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:05.360734 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:05.360735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:05.360814 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:05.360817 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:03:05.360875 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 03:03:05.361129 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":40,"threads_started":1}
I20260924 03:03:05.361227 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bda2b4926ae4d34addcfb0045d7cc63" candidate_uuid: "eb6bbbb5b66f46edbffd1964a617489a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6c64a32d2274811a9476856840a6e1b" is_pre_election: true
I20260924 03:03:05.361342 8062 raft_consensus.cc:2454] T 3bda2b4926ae4d34addcfb0045d7cc63 P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6bbbb5b66f46edbffd1964a617489a in term 0.
I20260924 03:03:05.361466 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bda2b4926ae4d34addcfb0045d7cc63" candidate_uuid: "eb6bbbb5b66f46edbffd1964a617489a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1407f59074514c53a31262544bbf537f" is_pre_election: true
I20260924 03:03:05.361573 8357 raft_consensus.cc:2454] T 3bda2b4926ae4d34addcfb0045d7cc63 P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6bbbb5b66f46edbffd1964a617489a in term 0.
I20260924 03:03:05.361524 8151 leader_election.cc:302] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6c64a32d2274811a9476856840a6e1b, eb6bbbb5b66f46edbffd1964a617489a; no voters:
I20260924 03:03:05.361663 8439 raft_consensus.cc:2781] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:03:05.361722 8439 raft_consensus.cc:482] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:03:05.361747 8439 raft_consensus.cc:3037] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:03:05.362422 8439 raft_consensus.cc:504] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:05.362560 8439 leader_election.cc:288] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a [CANDIDATE]: Term 1 election: Requested vote from peers b6c64a32d2274811a9476856840a6e1b (127.0.98.1:35847), 1407f59074514c53a31262544bbf537f (127.0.98.3:39147)
I20260924 03:03:05.362694 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bda2b4926ae4d34addcfb0045d7cc63" candidate_uuid: "eb6bbbb5b66f46edbffd1964a617489a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6c64a32d2274811a9476856840a6e1b"
I20260924 03:03:05.362739 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bda2b4926ae4d34addcfb0045d7cc63" candidate_uuid: "eb6bbbb5b66f46edbffd1964a617489a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1407f59074514c53a31262544bbf537f"
I20260924 03:03:05.362787 8062 raft_consensus.cc:3037] T 3bda2b4926ae4d34addcfb0045d7cc63 P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:03:05.362795 8357 raft_consensus.cc:3037] T 3bda2b4926ae4d34addcfb0045d7cc63 P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:03:05.363520 8062 raft_consensus.cc:2454] T 3bda2b4926ae4d34addcfb0045d7cc63 P b6c64a32d2274811a9476856840a6e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6bbbb5b66f46edbffd1964a617489a in term 1.
I20260924 03:03:05.363511 8357 raft_consensus.cc:2454] T 3bda2b4926ae4d34addcfb0045d7cc63 P 1407f59074514c53a31262544bbf537f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6bbbb5b66f46edbffd1964a617489a in term 1.
I20260924 03:03:05.363723 8151 leader_election.cc:302] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1407f59074514c53a31262544bbf537f, eb6bbbb5b66f46edbffd1964a617489a; no voters:
I20260924 03:03:05.363926 8439 raft_consensus.cc:2781] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:03:05.364054 8439 raft_consensus.cc:686] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a [term 1 LEADER]: Becoming Leader. State: Replica: eb6bbbb5b66f46edbffd1964a617489a, State: Running, Role: LEADER
I20260924 03:03:05.364125 8439 consensus_queue.cc:237] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:05.364830 7920 catalog_manager.cc:5495] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a reported cstate change: term changed from 0 to 1, leader changed from <none> to eb6bbbb5b66f46edbffd1964a617489a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "eb6bbbb5b66f46edbffd1964a617489a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } health_report { overall_health: UNKNOWN } } }
I20260924 03:03:05.390062 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:05.387568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51922 (local address 127.0.98.2:39853)
0924 03:03:05.387728 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:05.387729 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:05.387737 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:03:05.387773 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:05.387773 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:05.387791 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:05.387857 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:03:05.387979 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:05.388470 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:05.389223 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:05.389410 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:05.389458 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:05.389649 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:03:05.389759 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:03:05.389841 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:05.389876 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 03:03:05.389896 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":104,"threads_started":1}
I20260924 03:03:05.391253 8357 raft_consensus.cc:1261] T 3bda2b4926ae4d34addcfb0045d7cc63 P 1407f59074514c53a31262544bbf537f [term 1 FOLLOWER]: Refusing update from remote peer eb6bbbb5b66f46edbffd1964a617489a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:03:05.391273 8062 raft_consensus.cc:1261] T 3bda2b4926ae4d34addcfb0045d7cc63 P b6c64a32d2274811a9476856840a6e1b [term 1 FOLLOWER]: Refusing update from remote peer eb6bbbb5b66f46edbffd1964a617489a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:03:05.391671 8439 consensus_queue.cc:1035] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:03:05.391870 8505 consensus_queue.cc:1035] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:03:05.393512 8444 mvcc.cc:205] Tried to move back new op lower bound from 7332736964159889408 to 7332736964051943424. Current Snapshot: MvccSnapshot[applied={T|T < 7332736964159889408}]
I20260924 03:03:05.395152 7918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:03:05.395258 7918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:03:05.395932 7918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22ff55de1be043bd9bee0e8586e561d8: Sending DeleteTablet for 3 replicas of tablet 3bda2b4926ae4d34addcfb0045d7cc63
I20260924 03:03:05.396173 8057 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bda2b4926ae4d34addcfb0045d7cc63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:03:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59238
I20260924 03:03:05.396211 8197 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bda2b4926ae4d34addcfb0045d7cc63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:03:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51912
I20260924 03:03:05.396260 8337 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bda2b4926ae4d34addcfb0045d7cc63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:03:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40402
I20260924 03:03:05.396277 8454 tablet_replica.cc:330] T 3bda2b4926ae4d34addcfb0045d7cc63 P b6c64a32d2274811a9476856840a6e1b: stopping tablet replica
I20260924 03:03:05.396335 8454 raft_consensus.cc:2229] T 3bda2b4926ae4d34addcfb0045d7cc63 P b6c64a32d2274811a9476856840a6e1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:03:05.396342 8456 tablet_replica.cc:330] T 3bda2b4926ae4d34addcfb0045d7cc63 P 1407f59074514c53a31262544bbf537f: stopping tablet replica
I20260924 03:03:05.396396 8456 raft_consensus.cc:2229] T 3bda2b4926ae4d34addcfb0045d7cc63 P 1407f59074514c53a31262544bbf537f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:03:05.396389 8454 raft_consensus.cc:2258] T 3bda2b4926ae4d34addcfb0045d7cc63 P b6c64a32d2274811a9476856840a6e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:03:05.396435 8456 raft_consensus.cc:2258] T 3bda2b4926ae4d34addcfb0045d7cc63 P 1407f59074514c53a31262544bbf537f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:03:05.396484 8455 tablet_replica.cc:330] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a: stopping tablet replica
I20260924 03:03:05.396553 8455 raft_consensus.cc:2229] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:03:05.396647 8455 raft_consensus.cc:2258] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:03:05.396744 8456 ts_tablet_manager.cc:1905] T 3bda2b4926ae4d34addcfb0045d7cc63 P 1407f59074514c53a31262544bbf537f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:03:05.397065 7918 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 03:03:05.397150 8454 ts_tablet_manager.cc:1905] T 3bda2b4926ae4d34addcfb0045d7cc63 P b6c64a32d2274811a9476856840a6e1b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 03:03:05.397186 7918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:03:05.398334 8456 ts_tablet_manager.cc:1918] T 3bda2b4926ae4d34addcfb0045d7cc63 P 1407f59074514c53a31262544bbf537f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:03:05.398420 8456 log.cc:1196] T 3bda2b4926ae4d34addcfb0045d7cc63 P 1407f59074514c53a31262544bbf537f: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-2/wal/wals/3bda2b4926ae4d34addcfb0045d7cc63
I20260924 03:03:05.398661 8456 ts_tablet_manager.cc:1939] T 3bda2b4926ae4d34addcfb0045d7cc63 P 1407f59074514c53a31262544bbf537f: Deleting consensus metadata
I20260924 03:03:05.398743 8454 ts_tablet_manager.cc:1918] T 3bda2b4926ae4d34addcfb0045d7cc63 P b6c64a32d2274811a9476856840a6e1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:03:05.398803 8454 log.cc:1196] T 3bda2b4926ae4d34addcfb0045d7cc63 P b6c64a32d2274811a9476856840a6e1b: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-0/wal/wals/3bda2b4926ae4d34addcfb0045d7cc63
I20260924 03:03:05.398818 8197 tablet_service.cc:1467] Processing CreateTablet for tablet 8048b86f4a2948159d3a316233ad796a (DEFAULT_TABLE table=test-table [id=f581e8f04d484d999fabc06372363c82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:03:05.398921 8197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8048b86f4a2948159d3a316233ad796a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:03:05.399060 8454 ts_tablet_manager.cc:1939] T 3bda2b4926ae4d34addcfb0045d7cc63 P b6c64a32d2274811a9476856840a6e1b: Deleting consensus metadata
I20260924 03:03:05.399411 7907 catalog_manager.cc:4841] TS b6c64a32d2274811a9476856840a6e1b (127.0.98.1:35847): tablet 3bda2b4926ae4d34addcfb0045d7cc63 (table test-table [id=33f0fde9437f41d38ceca69686fcbc52]) successfully deleted
I20260924 03:03:05.399719 7907 catalog_manager.cc:4841] TS 1407f59074514c53a31262544bbf537f (127.0.98.3:39147): tablet 3bda2b4926ae4d34addcfb0045d7cc63 (table test-table [id=33f0fde9437f41d38ceca69686fcbc52]) successfully deleted
I20260924 03:03:05.399817 8337 tablet_service.cc:1467] Processing CreateTablet for tablet 8048b86f4a2948159d3a316233ad796a (DEFAULT_TABLE table=test-table [id=f581e8f04d484d999fabc06372363c82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:03:05.399825 8057 tablet_service.cc:1467] Processing CreateTablet for tablet 8048b86f4a2948159d3a316233ad796a (DEFAULT_TABLE table=test-table [id=f581e8f04d484d999fabc06372363c82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:03:05.399907 8057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8048b86f4a2948159d3a316233ad796a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:03:05.399907 8337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8048b86f4a2948159d3a316233ad796a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:03:05.401041 8431 tablet_bootstrap.cc:492] T 8048b86f4a2948159d3a316233ad796a P 1407f59074514c53a31262544bbf537f: Bootstrap starting.
I20260924 03:03:05.401064 8432 tablet_bootstrap.cc:492] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b: Bootstrap starting.
I20260924 03:03:05.401346 8432 tablet_bootstrap.cc:654] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b: Neither blocks nor log segments found. Creating new log.
I20260924 03:03:05.401391 8431 tablet_bootstrap.cc:654] T 8048b86f4a2948159d3a316233ad796a P 1407f59074514c53a31262544bbf537f: Neither blocks nor log segments found. Creating new log.
I20260924 03:03:05.401844 8432 tablet_bootstrap.cc:492] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b: No bootstrap required, opened a new log
I20260924 03:03:05.401911 8432 ts_tablet_manager.cc:1397] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 03:03:05.402022 8432 raft_consensus.cc:348] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:05.402093 8432 raft_consensus.cc:374] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:03:05.402127 8432 raft_consensus.cc:729] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6c64a32d2274811a9476856840a6e1b, State: Initialized, Role: FOLLOWER
I20260924 03:03:05.402174 8432 consensus_queue.cc:260] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:05.402263 8432 ts_tablet_manager.cc:1428] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:05.402911 8437 tablet_bootstrap.cc:492] T 8048b86f4a2948159d3a316233ad796a P eb6bbbb5b66f46edbffd1964a617489a: Bootstrap starting.
I20260924 03:03:05.403214 8455 ts_tablet_manager.cc:1905] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:03:05.403368 8437 tablet_bootstrap.cc:654] T 8048b86f4a2948159d3a316233ad796a P eb6bbbb5b66f46edbffd1964a617489a: Neither blocks nor log segments found. Creating new log.
I20260924 03:03:05.404122 8437 tablet_bootstrap.cc:492] T 8048b86f4a2948159d3a316233ad796a P eb6bbbb5b66f46edbffd1964a617489a: No bootstrap required, opened a new log
I20260924 03:03:05.404186 8437 ts_tablet_manager.cc:1397] T 8048b86f4a2948159d3a316233ad796a P eb6bbbb5b66f46edbffd1964a617489a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 03:03:05.404341 8437 raft_consensus.cc:348] T 8048b86f4a2948159d3a316233ad796a P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:05.404439 8437 raft_consensus.cc:374] T 8048b86f4a2948159d3a316233ad796a P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:03:05.404476 8437 raft_consensus.cc:729] T 8048b86f4a2948159d3a316233ad796a P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6bbbb5b66f46edbffd1964a617489a, State: Initialized, Role: FOLLOWER
I20260924 03:03:05.404541 8437 consensus_queue.cc:260] T 8048b86f4a2948159d3a316233ad796a P eb6bbbb5b66f46edbffd1964a617489a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:05.404572 8455 ts_tablet_manager.cc:1918] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:03:05.404625 8455 log.cc:1196] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-1/wal/wals/3bda2b4926ae4d34addcfb0045d7cc63
I20260924 03:03:05.404651 8437 ts_tablet_manager.cc:1428] T 8048b86f4a2948159d3a316233ad796a P eb6bbbb5b66f46edbffd1964a617489a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 03:03:05.404862 8455 ts_tablet_manager.cc:1939] T 3bda2b4926ae4d34addcfb0045d7cc63 P eb6bbbb5b66f46edbffd1964a617489a: Deleting consensus metadata
I20260924 03:03:05.405248 7907 catalog_manager.cc:4841] TS eb6bbbb5b66f46edbffd1964a617489a (127.0.98.2:39853): tablet 3bda2b4926ae4d34addcfb0045d7cc63 (table test-table [id=33f0fde9437f41d38ceca69686fcbc52]) successfully deleted
I20260924 03:03:05.405875 8431 tablet_bootstrap.cc:492] T 8048b86f4a2948159d3a316233ad796a P 1407f59074514c53a31262544bbf537f: No bootstrap required, opened a new log
I20260924 03:03:05.405941 8431 ts_tablet_manager.cc:1397] T 8048b86f4a2948159d3a316233ad796a P 1407f59074514c53a31262544bbf537f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260924 03:03:05.406072 8431 raft_consensus.cc:348] T 8048b86f4a2948159d3a316233ad796a P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:05.406162 8431 raft_consensus.cc:374] T 8048b86f4a2948159d3a316233ad796a P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:03:05.406188 8431 raft_consensus.cc:729] T 8048b86f4a2948159d3a316233ad796a P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1407f59074514c53a31262544bbf537f, State: Initialized, Role: FOLLOWER
I20260924 03:03:05.406236 8431 consensus_queue.cc:260] T 8048b86f4a2948159d3a316233ad796a P 1407f59074514c53a31262544bbf537f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:05.406347 8431 ts_tablet_manager.cc:1428] T 8048b86f4a2948159d3a316233ad796a P 1407f59074514c53a31262544bbf537f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:05.496225 8462 raft_consensus.cc:482] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:03:05.496342 8462 raft_consensus.cc:504] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:05.496497 8462 leader_election.cc:288] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6bbbb5b66f46edbffd1964a617489a (127.0.98.2:39853), 1407f59074514c53a31262544bbf537f (127.0.98.3:39147)
I20260924 03:03:05.496714 8216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8048b86f4a2948159d3a316233ad796a" candidate_uuid: "b6c64a32d2274811a9476856840a6e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6bbbb5b66f46edbffd1964a617489a" is_pre_election: true
I20260924 03:03:05.496801 8216 raft_consensus.cc:2454] T 8048b86f4a2948159d3a316233ad796a P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6c64a32d2274811a9476856840a6e1b in term 0.
I20260924 03:03:05.496716 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8048b86f4a2948159d3a316233ad796a" candidate_uuid: "b6c64a32d2274811a9476856840a6e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1407f59074514c53a31262544bbf537f" is_pre_election: true
I20260924 03:03:05.496851 8357 raft_consensus.cc:2454] T 8048b86f4a2948159d3a316233ad796a P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6c64a32d2274811a9476856840a6e1b in term 0.
I20260924 03:03:05.496959 8011 leader_election.cc:302] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6c64a32d2274811a9476856840a6e1b, eb6bbbb5b66f46edbffd1964a617489a; no voters:
I20260924 03:03:05.497076 8462 raft_consensus.cc:2781] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:03:05.497130 8462 raft_consensus.cc:482] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:03:05.497152 8462 raft_consensus.cc:3037] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:03:05.497854 8462 raft_consensus.cc:504] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:05.497980 8462 leader_election.cc:288] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b [CANDIDATE]: Term 1 election: Requested vote from peers eb6bbbb5b66f46edbffd1964a617489a (127.0.98.2:39853), 1407f59074514c53a31262544bbf537f (127.0.98.3:39147)
I20260924 03:03:05.498184 8216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8048b86f4a2948159d3a316233ad796a" candidate_uuid: "b6c64a32d2274811a9476856840a6e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6bbbb5b66f46edbffd1964a617489a"
I20260924 03:03:05.498261 8216 raft_consensus.cc:3037] T 8048b86f4a2948159d3a316233ad796a P eb6bbbb5b66f46edbffd1964a617489a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:03:05.499121 8216 raft_consensus.cc:2454] T 8048b86f4a2948159d3a316233ad796a P eb6bbbb5b66f46edbffd1964a617489a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6c64a32d2274811a9476856840a6e1b in term 1.
I20260924 03:03:05.499235 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8048b86f4a2948159d3a316233ad796a" candidate_uuid: "b6c64a32d2274811a9476856840a6e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1407f59074514c53a31262544bbf537f"
I20260924 03:03:05.499311 8357 raft_consensus.cc:3037] T 8048b86f4a2948159d3a316233ad796a P 1407f59074514c53a31262544bbf537f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:03:05.499310 8011 leader_election.cc:302] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6c64a32d2274811a9476856840a6e1b, eb6bbbb5b66f46edbffd1964a617489a; no voters:
I20260924 03:03:05.499392 8462 raft_consensus.cc:2781] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:03:05.499435 8462 raft_consensus.cc:686] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b [term 1 LEADER]: Becoming Leader. State: Replica: b6c64a32d2274811a9476856840a6e1b, State: Running, Role: LEADER
I20260924 03:03:05.499491 8462 consensus_queue.cc:237] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } }
I20260924 03:03:05.500093 8357 raft_consensus.cc:2454] T 8048b86f4a2948159d3a316233ad796a P 1407f59074514c53a31262544bbf537f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6c64a32d2274811a9476856840a6e1b in term 1.
I20260924 03:03:05.500139 7918 catalog_manager.cc:5495] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b reported cstate change: term changed from 0 to 1, leader changed from <none> to b6c64a32d2274811a9476856840a6e1b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b6c64a32d2274811a9476856840a6e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c64a32d2274811a9476856840a6e1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 } health_report { overall_health: UNKNOWN } } }
I20260924 03:03:05.525182 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:05.523059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59250 (local address 127.0.98.1:35847)
0924 03:03:05.523109 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:05.523111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:05.523196 (+ 85us) server_negotiation.cc:407] Connection header received
0924 03:03:05.523239 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:05.523239 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:05.523250 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:05.523294 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:03:05.523377 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:05.523915 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:05.524575 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:05.524749 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:05.524797 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:05.524900 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:03:05.524922 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:03:05.524955 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:05.525061 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 03:03:05.525077 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 03:03:05.527644 8216 raft_consensus.cc:1261] T 8048b86f4a2948159d3a316233ad796a P eb6bbbb5b66f46edbffd1964a617489a [term 1 FOLLOWER]: Refusing update from remote peer b6c64a32d2274811a9476856840a6e1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:03:05.527674 8357 raft_consensus.cc:1261] T 8048b86f4a2948159d3a316233ad796a P 1407f59074514c53a31262544bbf537f [term 1 FOLLOWER]: Refusing update from remote peer b6c64a32d2274811a9476856840a6e1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:03:05.527845 8462 consensus_queue.cc:1035] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6bbbb5b66f46edbffd1964a617489a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:03:05.527959 8436 consensus_queue.cc:1035] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1407f59074514c53a31262544bbf537f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:03:05.537976 7918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:03:05.538127 7918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:03:05.538736 7918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22ff55de1be043bd9bee0e8586e561d8: Sending DeleteTablet for 3 replicas of tablet 8048b86f4a2948159d3a316233ad796a
I20260924 03:03:05.538944 8057 tablet_service.cc:1514] Processing DeleteTablet for tablet 8048b86f4a2948159d3a316233ad796a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:03:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59238
I20260924 03:03:05.539011 8337 tablet_service.cc:1514] Processing DeleteTablet for tablet 8048b86f4a2948159d3a316233ad796a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:03:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40402
I20260924 03:03:05.539098 8456 tablet_replica.cc:330] T 8048b86f4a2948159d3a316233ad796a P 1407f59074514c53a31262544bbf537f: stopping tablet replica
I20260924 03:03:05.539098 8454 tablet_replica.cc:330] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b: stopping tablet replica
I20260924 03:03:05.539150 8456 raft_consensus.cc:2229] T 8048b86f4a2948159d3a316233ad796a P 1407f59074514c53a31262544bbf537f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:03:05.539150 8454 raft_consensus.cc:2229] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:03:05.539188 8456 raft_consensus.cc:2258] T 8048b86f4a2948159d3a316233ad796a P 1407f59074514c53a31262544bbf537f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:03:05.539211 8454 raft_consensus.cc:2258] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:03:05.539384 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 7992
I20260924 03:03:05.539551 8456 ts_tablet_manager.cc:1905] T 8048b86f4a2948159d3a316233ad796a P 1407f59074514c53a31262544bbf537f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:03:05.539548 8454 ts_tablet_manager.cc:1905] T 8048b86f4a2948159d3a316233ad796a P b6c64a32d2274811a9476856840a6e1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:03:05.541360 8456 ts_tablet_manager.cc:1918] T 8048b86f4a2948159d3a316233ad796a P 1407f59074514c53a31262544bbf537f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:03:05.541426 8456 log.cc:1196] T 8048b86f4a2948159d3a316233ad796a P 1407f59074514c53a31262544bbf537f: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218948286999-392-0/minicluster-data/ts-2/wal/wals/8048b86f4a2948159d3a316233ad796a
I20260924 03:03:05.541657 8456 ts_tablet_manager.cc:1939] T 8048b86f4a2948159d3a316233ad796a P 1407f59074514c53a31262544bbf537f: Deleting consensus metadata
I20260924 03:03:05.542158 7907 catalog_manager.cc:4841] TS 1407f59074514c53a31262544bbf537f (127.0.98.3:39147): tablet 8048b86f4a2948159d3a316233ad796a (table test-table [id=f581e8f04d484d999fabc06372363c82]) successfully deleted
I20260924 03:03:05.546151 8197 tablet_service.cc:1514] Processing DeleteTablet for tablet 8048b86f4a2948159d3a316233ad796a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:03:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51912
W20260924 03:03:05.546231 7907 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35847 (error 108)
I20260924 03:03:05.546255 8455 tablet_replica.cc:330] T 8048b86f4a2948159d3a316233ad796a P eb6bbbb5b66f46edbffd1964a617489a: stopping tablet replica
I20260924 03:03:05.546305 8455 raft_consensus.cc:2229] T 8048b86f4a2948159d3a316233ad796a P eb6bbbb5b66f46edbffd1964a617489a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:03:05.546355 8455 raft_consensus.cc:2258] T 8048b86f4a2948159d3a316233ad796a P eb6bbbb5b66f46edbffd1964a617489a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:03:05.546702 8455 ts_tablet_manager.cc:1905] T 8048b86f4a2948159d3a316233ad796a P eb6bbbb5b66f46edbffd1964a617489a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:03:05.546797 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:05.546598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35847 (local address 127.0.0.1:59262)
0924 03:03:05.546708 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:05.546755 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":54,"threads_started":1}
I20260924 03:03:05.546789 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 8132
W20260924 03:03:05.546974 7907 catalog_manager.cc:4584] TS b6c64a32d2274811a9476856840a6e1b (127.0.98.1:35847): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8048b86f4a2948159d3a316233ad796a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35847: connect: Connection refused (error 111)
I20260924 03:03:05.552947 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 8272
I20260924 03:03:05.552982 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:05.552841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39853 (local address 127.0.0.1:51938)
0924 03:03:05.552911 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:05.552943 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 03:03:05.558450 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 7888
2026-09-24T03:03:05Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3044 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:03:05 dist-test-slave-wgsb krb5kdc[8523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:03:05 dist-test-slave-wgsb krb5kdc[8523](info): set up 2 sockets
Sep 24 03:03:05 dist-test-slave-wgsb krb5kdc[8523](info): commencing operation
krb5kdc: starting...
W20260924 03:03:06.612442 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:03:06 dist-test-slave-wgsb krb5kdc[8523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:03:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:03:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:03:06.731252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38973
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:03:06.812783 8536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:03:06.813078 8536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:03:06.813128 8536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:03:06.814343 8536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:03:06.814416 8536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:03:06.814441 8536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:03:06.814467 8536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:03:06.814497 8536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:03:06.816197 8536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:03:06.816469 8536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:03:06.816694 8536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:03:06.821550 8542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:06.821533 8544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:06.821533 8541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:06.821852 8536 server_base.cc:1034] running on GCE node
I20260924 03:03:06.822067 8536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:03:06.822317 8536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:03:06.823463 8536 hybrid_clock.cc:648] HybridClock initialized: now 1790218986823438 us; error 44 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T03:03:06Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1269 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:03:06 dist-test-slave-wgsb krb5kdc[8549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:03:06 dist-test-slave-wgsb krb5kdc[8549](info): set up 2 sockets
Sep 24 03:03:06 dist-test-slave-wgsb krb5kdc[8549](info): commencing operation
krb5kdc: starting...
W20260924 03:03:07.908294 392 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:03:07 dist-test-slave-wgsb krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:03:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:03:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:03:08.031324 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44821
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:03:08.113801 8562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:03:08.114049 8562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:03:08.114089 8562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:03:08.115286 8562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:03:08.115355 8562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:03:08.115446 8562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:03:08.115481 8562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:03:08.115509 8562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:03:08.117108 8562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:03:08.117379 8562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:03:08.117611 8562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:03:08.122780 8568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:08.122818 8562 server_base.cc:1034] running on GCE node
W20260924 03:03:08.122728 8567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:08.122728 8570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:08.123247 8562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:03:08.123548 8562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:03:08.124744 8562 hybrid_clock.cc:648] HybridClock initialized: now 1790218988124723 us; error 38 us; skew 500 ppm
Sep 24 03:03:08 dist-test-slave-wgsb krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:03:08.135100 8562 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:03:08.135733 8562 webserver.cc:460] Webserver started at http://127.0.98.62:41335/ using document root <none> and password file <none>
I20260924 03:03:08.136008 8562 fs_manager.cc:359] Metadata directory not provided
I20260924 03:03:08.136091 8562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:03:08.136261 8562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:03:08.137449 8562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "9c64562fa4ea4b988d7a33d0956833ef"
format_stamp: "Formatted at 2026-09-24 03:03:08 on dist-test-slave-wgsb"
server_key: "530e71b31c70e674ba14a0c6aec0da23"
server_key_iv: "1a63aab1f4da03b1f418667998e455c6"
server_key_version: "encryptionkey@0"
I20260924 03:03:08.137847 8562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "9c64562fa4ea4b988d7a33d0956833ef"
format_stamp: "Formatted at 2026-09-24 03:03:08 on dist-test-slave-wgsb"
server_key: "530e71b31c70e674ba14a0c6aec0da23"
server_key_iv: "1a63aab1f4da03b1f418667998e455c6"
server_key_version: "encryptionkey@0"
I20260924 03:03:08.139369 8562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:03:08.140394 8577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:08.140575 8562 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:03:08.140695 8562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "9c64562fa4ea4b988d7a33d0956833ef"
format_stamp: "Formatted at 2026-09-24 03:03:08 on dist-test-slave-wgsb"
server_key: "530e71b31c70e674ba14a0c6aec0da23"
server_key_iv: "1a63aab1f4da03b1f418667998e455c6"
server_key_version: "encryptionkey@0"
I20260924 03:03:08.140776 8562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:03:08.153451 8562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:03:08.156139 8562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:03:08.156327 8562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:03:08.160969 8562 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44821
I20260924 03:03:08.161019 8629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44821 every 8 connection(s)
I20260924 03:03:08.161576 8562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:03:08.162259 8630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:03:08.164561 8630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef: Bootstrap starting.
I20260924 03:03:08.165153 8630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef: Neither blocks nor log segments found. Creating new log.
I20260924 03:03:08.165452 8630 log.cc:824] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef: Log is configured to *not* fsync() on all Append() calls
I20260924 03:03:08.166033 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 8562
I20260924 03:03:08.166137 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:03:08.166252 8630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef: No bootstrap required, opened a new log
I20260924 03:03:08.166318 392 external_mini_cluster.cc:1428] Setting key 79245b99365acc5e903e8aec84eaf009
I20260924 03:03:08.168677 8630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c64562fa4ea4b988d7a33d0956833ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44821 } }
I20260924 03:03:08.168838 8630 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:03:08.168900 8630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c64562fa4ea4b988d7a33d0956833ef, State: Initialized, Role: FOLLOWER
I20260924 03:03:08.169039 8630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c64562fa4ea4b988d7a33d0956833ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44821 } }
I20260924 03:03:08.169124 8630 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:03:08.169170 8630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:03:08.169222 8630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:03:08.169739 8630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c64562fa4ea4b988d7a33d0956833ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44821 } }
I20260924 03:03:08.169863 8630 leader_election.cc:302] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c64562fa4ea4b988d7a33d0956833ef; no voters:
I20260924 03:03:08.170061 8630 leader_election.cc:288] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:03:08.170102 8635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:03:08.170207 8635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [term 1 LEADER]: Becoming Leader. State: Replica: 9c64562fa4ea4b988d7a33d0956833ef, State: Running, Role: LEADER
I20260924 03:03:08.170285 8635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c64562fa4ea4b988d7a33d0956833ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44821 } }
I20260924 03:03:08.170321 8630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:03:08.170723 8635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c64562fa4ea4b988d7a33d0956833ef. Latest consensus state: current_term: 1 leader_uuid: "9c64562fa4ea4b988d7a33d0956833ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c64562fa4ea4b988d7a33d0956833ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44821 } } }
I20260924 03:03:08.170732 8636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c64562fa4ea4b988d7a33d0956833ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c64562fa4ea4b988d7a33d0956833ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44821 } } }
I20260924 03:03:08.170904 8635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [sys.catalog]: This master's current role is: LEADER
I20260924 03:03:08.170912 8636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [sys.catalog]: This master's current role is: LEADER
I20260924 03:03:08.171149 8641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:03:08.171732 8641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:03:08.173389 8641 catalog_manager.cc:1347] Generated new cluster ID: eb32d9e93f89486e9a3d5d637441c20f
I20260924 03:03:08.173442 8641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:03:08 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:03:08.178875 8634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:08.166898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54860 (local address 127.0.98.62:44821)
0924 03:03:08.167080 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:08.167087 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:08.167111 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:03:08.168669 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:08.168676 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:08.168743 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:08.168896 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:08.169620 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:08.171225 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:08.172378 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:08.172979 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:08.173120 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:08.175528 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:08.175532 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:08.175533 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:03:08.175544 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:08.177302 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:08.177679 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:08.177682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:08.177682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:08.177787 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:08.177995 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:08.177996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:08.177996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:08.178247 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:08.178422 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:08.178488 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 03:03:08.178732 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260924 03:03:08.183755 8641 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:03:08.184461 8641 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:03:08.197154 8641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef: Generated new TSK 0
I20260924 03:03:08.197337 8641 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:03:08.235409 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44821
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:03:08.316586 8656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:03:08.316810 8656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:03:08.316834 8656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:03:08.318015 8656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:03:08.318063 8656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:03:08.318089 8656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:03:08.319697 8656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:03:08.319968 8656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:03:08.320190 8656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:03:08.325449 8662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:08.325455 8661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:08.325450 8664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:08.325532 8656 server_base.cc:1034] running on GCE node
I20260924 03:03:08.326017 8656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:03:08.326283 8656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:03:08.327450 8656 hybrid_clock.cc:648] HybridClock initialized: now 1790218988327448 us; error 43 us; skew 500 ppm
Sep 24 03:03:08 dist-test-slave-wgsb krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:03:08.330257 8656 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:03:08.330730 8656 webserver.cc:460] Webserver started at http://127.0.98.1:40123/ using document root <none> and password file <none>
I20260924 03:03:08.330967 8656 fs_manager.cc:359] Metadata directory not provided
I20260924 03:03:08.331012 8656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:03:08.331164 8656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:03:08.332300 8656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/data/instance:
uuid: "c1c1696348f74f23b61609848df4ebb3"
format_stamp: "Formatted at 2026-09-24 03:03:08 on dist-test-slave-wgsb"
server_key: "d12a5dbc804bef81f589ffcd579ae0b9"
server_key_iv: "498160b63e39d891314e939e492650ad"
server_key_version: "encryptionkey@0"
I20260924 03:03:08.332659 8656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c1c1696348f74f23b61609848df4ebb3"
format_stamp: "Formatted at 2026-09-24 03:03:08 on dist-test-slave-wgsb"
server_key: "d12a5dbc804bef81f589ffcd579ae0b9"
server_key_iv: "498160b63e39d891314e939e492650ad"
server_key_version: "encryptionkey@0"
I20260924 03:03:08.334156 8656 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 03:03:08.334969 8671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:08.335215 8656 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:03:08.335335 8656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "c1c1696348f74f23b61609848df4ebb3"
format_stamp: "Formatted at 2026-09-24 03:03:08 on dist-test-slave-wgsb"
server_key: "d12a5dbc804bef81f589ffcd579ae0b9"
server_key_iv: "498160b63e39d891314e939e492650ad"
server_key_version: "encryptionkey@0"
I20260924 03:03:08.335409 8656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:03:08.351038 8656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:03:08.353421 8656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:03:08.353586 8656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:03:08.353825 8656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:03:08.354126 8656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:03:08.354161 8656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:08.354188 8656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:03:08.354205 8656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:08.360229 8656 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41271
I20260924 03:03:08.360298 8784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41271 every 8 connection(s)
I20260924 03:03:08.360638 8656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 03:03:08 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:03:08.370432 8634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:08.361393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32777 (local address 127.0.98.62:44821)
0924 03:03:08.361447 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:08.361449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:08.361700 (+ 251us) server_negotiation.cc:407] Connection header received
0924 03:03:08.362903 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:08.362905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:08.362920 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:08.362975 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:08.363619 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:08.364130 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:08.365133 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:08.365354 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:08.365440 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:08.367957 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:08.367958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:08.367959 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:03:08.367965 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:08.369175 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:08.369538 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:08.369539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:08.369540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:08.369587 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:08.369867 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:08.369868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:08.369868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:08.369970 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:08.370024 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:08.370287 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 03:03:08.370355 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 03:03:08.370554 8787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:08.361392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.1:32777)
0924 03:03:08.361546 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:08.361586 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:08.362791 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:08.363046 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:08.363050 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:08.363315 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:08.363499 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:08.363503 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:08.364226 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:08.364227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:08.365034 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:08.365035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:08.365450 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:08.365451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:08.365587 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:08.366043 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:08.366051 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:08.367869 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:03:08.369247 (+ 1378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:03:08.369253 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:03:08.369255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:03:08.369459 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:03:08.369667 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:03:08.369668 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:03:08.369668 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:03:08.369801 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:03:08.370045 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:03:08.370051 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:03:08.370173 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 03:03:08.370257 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 03:03:08.370372 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260924 03:03:08.370971 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 8656
I20260924 03:03:08.371091 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:03:08.371203 8785 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44821
I20260924 03:03:08.371299 8785 heartbeater.cc:464] Registering TS with master...
I20260924 03:03:08.371368 392 external_mini_cluster.cc:1428] Setting key fb007796aa61c5abdfa3d5e77db0ca93
I20260924 03:03:08.371507 8785 heartbeater.cc:511] Master 127.0.98.62:44821 requested a full tablet report, sending...
I20260924 03:03:08.372073 8593 ts_manager.cc:194] Registered new tserver with Master: c1c1696348f74f23b61609848df4ebb3 (127.0.98.1:41271)
I20260924 03:03:08.372668 8593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:32777
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:03:08.398617 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44821
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:03:08.479779 8790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:03:08.480041 8790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:03:08.480077 8790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:03:08.481256 8790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:03:08.481323 8790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:03:08.481369 8790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:03:08.482959 8790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:03:08.483230 8790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:03:08.483472 8790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:03:08.489203 8796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:08.489259 8798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:08.489216 8795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:08.489583 8790 server_base.cc:1034] running on GCE node
I20260924 03:03:08.489859 8790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:03:08.490094 8790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:03:08.491292 8790 hybrid_clock.cc:648] HybridClock initialized: now 1790218988491278 us; error 26 us; skew 500 ppm
Sep 24 03:03:08 dist-test-slave-wgsb krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:03:08.494143 8790 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:03:08.494702 8790 webserver.cc:460] Webserver started at http://127.0.98.2:35153/ using document root <none> and password file <none>
I20260924 03:03:08.494959 8790 fs_manager.cc:359] Metadata directory not provided
I20260924 03:03:08.495033 8790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:03:08.495196 8790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:03:08.496385 8790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/data/instance:
uuid: "59768f7cfe514cea8e264ad4c692534b"
format_stamp: "Formatted at 2026-09-24 03:03:08 on dist-test-slave-wgsb"
server_key: "d22b31b29c34376ed71f6a4f8fc05b09"
server_key_iv: "fe715789e7a549360472e3019a92a275"
server_key_version: "encryptionkey@0"
I20260924 03:03:08.496797 8790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/wal/instance:
uuid: "59768f7cfe514cea8e264ad4c692534b"
format_stamp: "Formatted at 2026-09-24 03:03:08 on dist-test-slave-wgsb"
server_key: "d22b31b29c34376ed71f6a4f8fc05b09"
server_key_iv: "fe715789e7a549360472e3019a92a275"
server_key_version: "encryptionkey@0"
I20260924 03:03:08.498186 8790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:03:08.498977 8805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:08.499174 8790 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:03:08.499287 8790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "59768f7cfe514cea8e264ad4c692534b"
format_stamp: "Formatted at 2026-09-24 03:03:08 on dist-test-slave-wgsb"
server_key: "d22b31b29c34376ed71f6a4f8fc05b09"
server_key_iv: "fe715789e7a549360472e3019a92a275"
server_key_version: "encryptionkey@0"
I20260924 03:03:08.499373 8790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:03:08.522226 8790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:03:08.525126 8790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:03:08.525305 8790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:03:08.525621 8790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:03:08.526046 8790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:03:08.526093 8790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:08.526129 8790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:03:08.526192 8790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:08.533409 8790 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36813
I20260924 03:03:08.533445 8918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36813 every 8 connection(s)
I20260924 03:03:08.533798 8790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 03:03:08 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:03:08.542863 8634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:08.534563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45243 (local address 127.0.98.62:44821)
0924 03:03:08.534616 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:08.534619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:08.534848 (+ 229us) server_negotiation.cc:407] Connection header received
0924 03:03:08.535858 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:08.535859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:08.535869 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:08.535925 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:08.536483 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:08.536978 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:08.537929 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:08.538112 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:08.538213 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:08.540327 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:08.540328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:08.540328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:03:08.540335 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:08.541792 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:08.542131 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:08.542132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:08.542132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:08.542182 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:08.542378 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:08.542378 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:08.542378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:08.542475 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:08.542523 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:08.542739 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 03:03:08.542800 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:03:08.543081 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:08.534563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.2:45243)
0924 03:03:08.534732 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:08.534768 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:08.535750 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:08.535977 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:08.535980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:08.536227 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:08.536415 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:08.536419 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:08.537059 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:08.537060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:08.537857 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:08.537859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:08.538223 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:08.538224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:08.538372 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:08.538821 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:08.538829 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:08.540230 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:03:08.541858 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:03:08.541863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:03:08.541865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:03:08.542067 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:03:08.542237 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:03:08.542238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:03:08.542238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:03:08.542328 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:03:08.542523 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:03:08.542527 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:03:08.542633 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 03:03:08.542715 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 03:03:08.542831 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20260924 03:03:08.543535 8919 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44821
I20260924 03:03:08.543660 8919 heartbeater.cc:464] Registering TS with master...
I20260924 03:03:08.543823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 8790
I20260924 03:03:08.543869 8919 heartbeater.cc:511] Master 127.0.98.62:44821 requested a full tablet report, sending...
I20260924 03:03:08.543918 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:03:08.544122 392 external_mini_cluster.cc:1428] Setting key f8011b98b61e1d44fd354065a5ea7123
I20260924 03:03:08.544395 8593 ts_manager.cc:194] Registered new tserver with Master: 59768f7cfe514cea8e264ad4c692534b (127.0.98.2:36813)
I20260924 03:03:08.544795 8593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45243
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:03:08.571058 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44821
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:03:08.657826 8924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:03:08.658046 8924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:03:08.658082 8924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:03:08.659202 8924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:03:08.659262 8924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:03:08.659307 8924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:03:08.660822 8924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:03:08.661079 8924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:03:08.661309 8924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:03:08.666349 8929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:08.666373 8930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:08.666337 8932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:08.666623 8924 server_base.cc:1034] running on GCE node
I20260924 03:03:08.666877 8924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:03:08.667169 8924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:03:08.668336 8924 hybrid_clock.cc:648] HybridClock initialized: now 1790218988668320 us; error 32 us; skew 500 ppm
Sep 24 03:03:08 dist-test-slave-wgsb krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:03:08.671051 8924 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:03:08.671540 8924 webserver.cc:460] Webserver started at http://127.0.98.3:42033/ using document root <none> and password file <none>
I20260924 03:03:08.671789 8924 fs_manager.cc:359] Metadata directory not provided
I20260924 03:03:08.671835 8924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:03:08.671968 8924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:03:08.673070 8924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/data/instance:
uuid: "c98f0bc8a07f42198d63c2b31cfe18b0"
format_stamp: "Formatted at 2026-09-24 03:03:08 on dist-test-slave-wgsb"
server_key: "822fae34b899f72bd0eadd6ab2cba7c2"
server_key_iv: "2d5defada3b7846d25f5d89b8a0ecd71"
server_key_version: "encryptionkey@0"
I20260924 03:03:08.673369 8924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c98f0bc8a07f42198d63c2b31cfe18b0"
format_stamp: "Formatted at 2026-09-24 03:03:08 on dist-test-slave-wgsb"
server_key: "822fae34b899f72bd0eadd6ab2cba7c2"
server_key_iv: "2d5defada3b7846d25f5d89b8a0ecd71"
server_key_version: "encryptionkey@0"
I20260924 03:03:08.675848 8924 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 03:03:08.676578 8939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:08.676762 8924 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:03:08.676833 8924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "c98f0bc8a07f42198d63c2b31cfe18b0"
format_stamp: "Formatted at 2026-09-24 03:03:08 on dist-test-slave-wgsb"
server_key: "822fae34b899f72bd0eadd6ab2cba7c2"
server_key_iv: "2d5defada3b7846d25f5d89b8a0ecd71"
server_key_version: "encryptionkey@0"
I20260924 03:03:08.676887 8924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:03:08.708591 8924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:03:08.710901 8924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:03:08.711021 8924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:03:08.711215 8924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:03:08.711535 8924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:03:08.711570 8924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:08.711633 8924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:03:08.711663 8924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:08.717434 8924 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40225
I20260924 03:03:08.717516 9052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40225 every 8 connection(s)
I20260924 03:03:08.717808 8924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 03:03:08 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:03:08.726210 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 8924
I20260924 03:03:08.726315 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:03:08.726507 392 external_mini_cluster.cc:1428] Setting key a805841e92b3dd01fac0f74098e18de8
I20260924 03:03:08.728039 8634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:08.718584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35739 (local address 127.0.98.62:44821)
0924 03:03:08.718643 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:08.718645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:08.718875 (+ 230us) server_negotiation.cc:407] Connection header received
0924 03:03:08.719948 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:08.719949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:08.719959 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:08.720023 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:08.720656 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:08.721242 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:08.722317 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:08.722515 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:08.722594 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:08.724816 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:08.724818 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:08.724818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:03:08.724825 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:08.726571 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:08.726999 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:08.727000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:08.727000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:08.727060 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:08.727322 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:08.727323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:08.727323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:08.727458 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:08.727537 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:08.727862 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 03:03:08.727944 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 03:03:08.728255 9055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:08.718562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.3:35739)
0924 03:03:08.718751 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:08.718793 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:08.719829 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:08.720086 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:08.720089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:08.720373 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:08.720563 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:08.720567 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:08.721353 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:08.721354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:08.722200 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:08.722202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:08.722603 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:08.722604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:08.722729 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:08.723204 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:08.723212 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:08.724729 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:03:08.726654 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:03:08.726661 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:03:08.726663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:03:08.726935 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:03:08.727141 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:03:08.727142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:03:08.727142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:03:08.727256 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:03:08.727529 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:03:08.727535 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:03:08.727732 (+ 197us) client_negotiation.cc:769] Sending connection context
0924 03:03:08.727840 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 03:03:08.728006 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":119,"threads_started":1}
I20260924 03:03:08.728684 9053 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44821
I20260924 03:03:08.728782 9053 heartbeater.cc:464] Registering TS with master...
I20260924 03:03:08.728958 9053 heartbeater.cc:511] Master 127.0.98.62:44821 requested a full tablet report, sending...
I20260924 03:03:08.729314 8593 ts_manager.cc:194] Registered new tserver with Master: c98f0bc8a07f42198d63c2b31cfe18b0 (127.0.98.3:40225)
I20260924 03:03:08.729692 8593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35739
I20260924 03:03:08.738237 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 03:03:08.810571 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 8656
I20260924 03:03:08.814965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 8790
I20260924 03:03:08.819340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 8924
I20260924 03:03:08.823324 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 8562
I20260924 03:03:08.829181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44821
--webserver_interface=127.0.98.62
--webserver_port=41335
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:03:08.921869 9062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:03:08.922122 9062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:03:08.922161 9062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:03:08.923382 9062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:03:08.923454 9062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:03:08.923478 9062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:03:08.923507 9062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:03:08.923537 9062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:03:08.925132 9062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:03:08.925436 9062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:03:08.925665 9062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:03:08.930699 9068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:08.930716 9067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:08.930699 9070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:08.931082 9062 server_base.cc:1034] running on GCE node
I20260924 03:03:08.931397 9062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:03:08.931740 9062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:03:08.932891 9062 hybrid_clock.cc:648] HybridClock initialized: now 1790218988932858 us; error 51 us; skew 500 ppm
Sep 24 03:03:08 dist-test-slave-wgsb krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:03:08.935452 9062 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:03:08.935958 9062 webserver.cc:460] Webserver started at http://127.0.98.62:41335/ using document root <none> and password file <none>
I20260924 03:03:08.936147 9062 fs_manager.cc:359] Metadata directory not provided
I20260924 03:03:08.936198 9062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:03:08.937491 9062 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:03:08.938113 9077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:08.938321 9062 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:03:08.938416 9062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "9c64562fa4ea4b988d7a33d0956833ef"
format_stamp: "Formatted at 2026-09-24 03:03:08 on dist-test-slave-wgsb"
server_key: "530e71b31c70e674ba14a0c6aec0da23"
server_key_iv: "1a63aab1f4da03b1f418667998e455c6"
server_key_version: "encryptionkey@0"
I20260924 03:03:08.938807 9062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:03:08.950029 9062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:03:08.951895 9062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:03:08.952049 9062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:03:08.956007 9062 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44821
I20260924 03:03:08.956037 9129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44821 every 8 connection(s)
I20260924 03:03:08.956570 9062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:03:08.957857 9130 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:03:08.958500 9130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef: Bootstrap starting.
I20260924 03:03:08.964066 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 9062
I20260924 03:03:08.964407 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41271
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44821
--webserver_port=40123
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:03:08.966908 9130 log.cc:824] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef: Log is configured to *not* fsync() on all Append() calls
I20260924 03:03:08.969064 9130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:03:08.969358 9130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef: Bootstrap complete.
I20260924 03:03:08.971566 9130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c64562fa4ea4b988d7a33d0956833ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44821 } }
I20260924 03:03:08.971890 9130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c64562fa4ea4b988d7a33d0956833ef, State: Initialized, Role: FOLLOWER
I20260924 03:03:08.972060 9130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c64562fa4ea4b988d7a33d0956833ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44821 } }
I20260924 03:03:08.972129 9130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:03:08.972187 9130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:03:08.972236 9130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:03:08.973268 9130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c64562fa4ea4b988d7a33d0956833ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44821 } }
I20260924 03:03:08.973354 9130 leader_election.cc:302] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c64562fa4ea4b988d7a33d0956833ef; no voters:
I20260924 03:03:08.973582 9130 leader_election.cc:288] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 03:03:08.973610 9134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [term 2 FOLLOWER]: Leader election won for term 2
I20260924 03:03:08.973739 9134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [term 2 LEADER]: Becoming Leader. State: Replica: 9c64562fa4ea4b988d7a33d0956833ef, State: Running, Role: LEADER
I20260924 03:03:08.973891 9130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:03:08.973912 9134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c64562fa4ea4b988d7a33d0956833ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44821 } }
I20260924 03:03:08.974198 9134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c64562fa4ea4b988d7a33d0956833ef. Latest consensus state: current_term: 2 leader_uuid: "9c64562fa4ea4b988d7a33d0956833ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c64562fa4ea4b988d7a33d0956833ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44821 } } }
I20260924 03:03:08.974293 9134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [sys.catalog]: This master's current role is: LEADER
I20260924 03:03:08.974203 9136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9c64562fa4ea4b988d7a33d0956833ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c64562fa4ea4b988d7a33d0956833ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44821 } } }
I20260924 03:03:08.974478 9136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef [sys.catalog]: This master's current role is: LEADER
I20260924 03:03:08.974994 9140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:03:08.975507 9140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:03:08.975688 9140 catalog_manager.cc:1259] Loaded cluster ID: eb32d9e93f89486e9a3d5d637441c20f
I20260924 03:03:08.975740 9140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:03:08.976536 9140 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:03:08.976725 9140 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9c64562fa4ea4b988d7a33d0956833ef: Loaded TSK: 0
I20260924 03:03:08.976933 9140 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 03:03:09.059391 9132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:03:09.059581 9132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:03:09.059656 9132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:03:09.060807 9132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:03:09.060856 9132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:03:09.060887 9132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:03:09.062306 9132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40123
--tserver_master_addrs=127.0.98.62:44821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:03:09.062527 9132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:03:09.062763 9132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:03:09.067782 9157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:09.067762 9159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:09.067763 9156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:09.068316 9132 server_base.cc:1034] running on GCE node
I20260924 03:03:09.068522 9132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:03:09.068799 9132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:03:09.069952 9132 hybrid_clock.cc:648] HybridClock initialized: now 1790218989069932 us; error 35 us; skew 500 ppm
Sep 24 03:03:09 dist-test-slave-wgsb krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218989, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:03:09.072460 9132 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 03:03:09.072906 9132 webserver.cc:460] Webserver started at http://127.0.98.1:40123/ using document root <none> and password file <none>
I20260924 03:03:09.073134 9132 fs_manager.cc:359] Metadata directory not provided
I20260924 03:03:09.073189 9132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:03:09.074697 9132 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:03:09.075330 9166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:09.075502 9132 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:03:09.075584 9132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "c1c1696348f74f23b61609848df4ebb3"
format_stamp: "Formatted at 2026-09-24 03:03:08 on dist-test-slave-wgsb"
server_key: "d12a5dbc804bef81f589ffcd579ae0b9"
server_key_iv: "498160b63e39d891314e939e492650ad"
server_key_version: "encryptionkey@0"
I20260924 03:03:09.075973 9132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:03:09.091849 9132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:03:09.094142 9132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:03:09.094285 9132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:03:09.094520 9132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:03:09.094844 9132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:03:09.094878 9132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:09.094899 9132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:03:09.094913 9132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:09.101997 9132 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41271
I20260924 03:03:09.102038 9279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41271 every 8 connection(s)
I20260924 03:03:09.102403 9132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:03:09.109901 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 9132
I20260924 03:03:09.110229 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36813
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44821
--webserver_port=35153
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:03:09 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:09 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:09.112447 9282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:09.103190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.1:38907)
0924 03:03:09.103384 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:09.103429 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:09.104118 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:09.104730 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:09.104734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:09.105537 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:09.105825 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:09.105831 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.107509 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.107510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:09.108470 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:09.108473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.109115 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.109116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:09.109242 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:09.109793 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:09.109801 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:09.112224 (+ 2423us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
W20260924 03:03:09.112692 9280 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44821 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:44821: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260924 03:03:09.112828 9283 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:09.103313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38907 (local address 127.0.98.62:44821)
0924 03:03:09.103474 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:09.103482 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:09.103521 (+ 39us) server_negotiation.cc:407] Connection header received
0924 03:03:09.104408 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:09.104416 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:09.104501 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:09.104647 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:09.105968 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.107404 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.108594 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.108999 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.109135 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:09.112345 (+ 3210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38907: BlockingRecv error: recv got EOF from 127.0.98.1:38907 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
Sep 24 03:03:09 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:09 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:09.117415 9282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:09.112926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.1:56699)
0924 03:03:09.113157 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:09.113166 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:09.113240 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:09.113379 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:09.113380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:09.113514 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:09.113565 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:09.113567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.114198 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.114199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:09.114706 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:09.114707 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.115077 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.115078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:09.115155 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:09.115624 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:09.115630 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:09.117331 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217}
W20260924 03:03:09.117467 9283 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:09.113004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56699 (local address 127.0.98.62:44821)
0924 03:03:09.113042 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:09.113044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:09.113217 (+ 173us) server_negotiation.cc:407] Connection header received
0924 03:03:09.113300 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:09.113300 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:09.113318 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:09.113376 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:09.113642 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.114112 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.114798 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.114986 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.115075 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:09.117404 (+ 2329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56699: BlockingRecv error: recv got EOF from 127.0.98.1:56699 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Sep 24 03:03:09 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:09 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:09.121943 9282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:09.117905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.1:59351)
0924 03:03:09.117946 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:09.117956 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:09.117991 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:09.118155 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:09.118156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:09.118275 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:09.118301 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:09.118302 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.118910 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.118911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:09.119395 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:09.119396 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.119823 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.119824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:09.119877 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:09.120295 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:09.120300 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:09.121859 (+ 1559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 03:03:09.122228 9280 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:03:09.122336 9283 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:09.117949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59351 (local address 127.0.98.62:44821)
0924 03:03:09.118054 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:09.118055 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:09.118060 (+ 5us) server_negotiation.cc:407] Connection header received
0924 03:03:09.118077 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:09.118077 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:09.118085 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:09.118151 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:09.118377 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.118823 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.119488 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.119746 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.119823 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:09.121933 (+ 2110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59351: BlockingRecv error: recv got EOF from 127.0.98.1:59351 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260924 03:03:09.194171 9284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:03:09.194368 9284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:03:09.194391 9284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:03:09.195616 9284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:03:09.195657 9284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:03:09.195685 9284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:03:09.197002 9284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35153
--tserver_master_addrs=127.0.98.62:44821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:03:09.197190 9284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:03:09.197355 9284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:03:09.202165 9289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:09.202313 9290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:09.202425 9284 server_base.cc:1034] running on GCE node
W20260924 03:03:09.202431 9292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:09.202692 9284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:03:09.202931 9284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:03:09.204121 9284 hybrid_clock.cc:648] HybridClock initialized: now 1790218989204112 us; error 37 us; skew 500 ppm
Sep 24 03:03:09 dist-test-slave-wgsb krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218989, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:03:09.206611 9284 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 03:03:09.206983 9284 webserver.cc:460] Webserver started at http://127.0.98.2:35153/ using document root <none> and password file <none>
I20260924 03:03:09.207160 9284 fs_manager.cc:359] Metadata directory not provided
I20260924 03:03:09.207211 9284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:03:09.208505 9284 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:03:09.209009 9299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:09.209223 9284 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:03:09.209414 9284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "59768f7cfe514cea8e264ad4c692534b"
format_stamp: "Formatted at 2026-09-24 03:03:08 on dist-test-slave-wgsb"
server_key: "d22b31b29c34376ed71f6a4f8fc05b09"
server_key_iv: "fe715789e7a549360472e3019a92a275"
server_key_version: "encryptionkey@0"
I20260924 03:03:09.209762 9284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:03:09.227514 9284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:03:09.229887 9284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:03:09.230039 9284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:03:09.230305 9284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:03:09.230618 9284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:03:09.230651 9284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:09.230692 9284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:03:09.230719 9284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:09.236639 9284 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36813
I20260924 03:03:09.236729 9412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36813 every 8 connection(s)
I20260924 03:03:09.237046 9284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 03:03:09 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:09 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:09.244304 9415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:09.237794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.2:57225)
0924 03:03:09.237943 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:09.237977 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:09.238462 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:09.238741 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:09.238744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:09.239299 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:09.239484 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:09.239488 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.240226 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.240226 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:09.241058 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:09.241060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.241373 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.241373 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:09.241468 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:09.241981 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:09.241990 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:09.244158 (+ 2168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
W20260924 03:03:09.244311 9283 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:09.237786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57225 (local address 127.0.98.62:44821)
0924 03:03:09.237837 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:09.237839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:09.238057 (+ 218us) server_negotiation.cc:407] Connection header received
0924 03:03:09.238590 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:09.238592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:09.238605 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:09.238651 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:09.239621 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.240118 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.241126 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.241304 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.241361 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:09.244265 (+ 2904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57225: BlockingRecv error: recv got EOF from 127.0.98.2:57225 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 03:03:09.244561 9413 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44821 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:44821: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260924 03:03:09.246251 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 9284
I20260924 03:03:09.246553 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40225
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44821
--webserver_port=42033
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:03:09 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:09 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:09.249634 9415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:09.244777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.2:38717)
0924 03:03:09.244832 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:09.244844 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:09.244890 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:09.245001 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:09.245002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:09.245167 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:09.245211 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:09.245213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.245836 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.245837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:09.246680 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:09.246682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.247080 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.247081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:09.247160 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:09.247564 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:09.247570 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:09.249585 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 03:03:09.249774 9283 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:09.244845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38717 (local address 127.0.98.62:44821)
0924 03:03:09.244896 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:09.244897 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:09.244922 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:03:09.244943 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:09.244943 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:09.244951 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:09.244989 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:09.245272 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.245736 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.246806 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.247002 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.247063 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:09.249691 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38717: BlockingRecv error: recv got EOF from 127.0.98.2:38717 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 24 03:03:09 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:09 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:09.254528 9415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:09.249934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.2:58091)
0924 03:03:09.250038 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:09.250049 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:09.250089 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:09.250204 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:09.250204 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:09.250354 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:09.250384 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:09.250385 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.251307 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.251308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:09.251826 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:09.251828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.252304 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.252305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:09.252339 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:09.252742 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:09.252746 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:09.254483 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89}
W20260924 03:03:09.254673 9283 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:09.250032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58091 (local address 127.0.98.62:44821)
0924 03:03:09.250088 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:09.250090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:09.250100 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:03:09.250126 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:09.250127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:09.250138 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:09.250198 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:09.250452 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.251232 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.251896 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.252230 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.252314 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:09.254589 (+ 2275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58091: BlockingRecv error: recv got EOF from 127.0.98.2:58091 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260924 03:03:09.254660 9413 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:03:09.328969 9416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:03:09.329160 9416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:03:09.329191 9416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:03:09.330291 9416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:03:09.330327 9416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:03:09.330353 9416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:03:09.331756 9416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40225
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42033
--tserver_master_addrs=127.0.98.62:44821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:03:09.331941 9416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:03:09.332111 9416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:03:09.336865 9424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:09.336898 9422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:09.336861 9421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:09.337132 9416 server_base.cc:1034] running on GCE node
I20260924 03:03:09.337355 9416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:03:09.337589 9416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:03:09.338752 9416 hybrid_clock.cc:648] HybridClock initialized: now 1790218989338731 us; error 42 us; skew 500 ppm
Sep 24 03:03:09 dist-test-slave-wgsb krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218989, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:03:09.341398 9416 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 03:03:09.341825 9416 webserver.cc:460] Webserver started at http://127.0.98.3:42033/ using document root <none> and password file <none>
I20260924 03:03:09.342067 9416 fs_manager.cc:359] Metadata directory not provided
I20260924 03:03:09.342123 9416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:03:09.343494 9416 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:03:09.344152 9431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:09.344355 9416 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 03:03:09.344447 9416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "c98f0bc8a07f42198d63c2b31cfe18b0"
format_stamp: "Formatted at 2026-09-24 03:03:08 on dist-test-slave-wgsb"
server_key: "822fae34b899f72bd0eadd6ab2cba7c2"
server_key_iv: "2d5defada3b7846d25f5d89b8a0ecd71"
server_key_version: "encryptionkey@0"
I20260924 03:03:09.344803 9416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:03:09.360191 9416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:03:09.362263 9416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:03:09.362427 9416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:03:09.362653 9416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:03:09.362955 9416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:03:09.362988 9416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:09.363010 9416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:03:09.363023 9416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:09.369258 9416 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40225
I20260924 03:03:09.369377 9544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40225 every 8 connection(s)
I20260924 03:03:09.369664 9416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:03:09.372225 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 9416
Sep 24 03:03:09 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:09 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 03:03:09.377667 9283 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:09.370424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41585 (local address 127.0.98.62:44821)
0924 03:03:09.370513 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:09.370515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:09.370699 (+ 184us) server_negotiation.cc:407] Connection header received
0924 03:03:09.371296 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:09.371297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:09.371309 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:09.371374 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:09.372356 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.372978 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.374055 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.374269 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.374365 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:09.377609 (+ 3244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41585: BlockingRecv error: recv got EOF from 127.0.98.3:41585 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 03:03:09.377763 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:09.370328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.3:41585)
0924 03:03:09.370530 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:09.370579 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:09.371095 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:09.371443 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:09.371447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:09.372029 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:09.372227 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:09.372231 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.373075 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.373076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:09.373975 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:09.373978 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.374378 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.374379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:09.374500 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:09.375037 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:09.375059 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:09.377555 (+ 2496us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":102,"threads_started":1}
W20260924 03:03:09.377933 9545 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44821 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:44821: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260924 03:03:09.379961 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:09.372729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54870 (local address 127.0.98.62:44821)
0924 03:03:09.372842 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:09.372844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:09.372852 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:03:09.372877 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:09.372878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:09.372887 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:09.372954 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:09.373596 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.374214 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.375055 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.375263 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.375331 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:09.376646 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:09.376650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:09.376650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:03:09.376660 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:09.378472 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:09.378935 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:09.378938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:09.378938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:09.379034 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:09.379236 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:09.379237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:09.379238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:09.379453 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:09.379595 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:09.379744 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 03:03:09.379875 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
Sep 24 03:03:09 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:09 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:09.382474 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:09.378156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.3:41211)
0924 03:03:09.378257 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:09.378267 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:09.378311 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:09.378434 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:09.378435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:09.378556 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:09.378595 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:09.378597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.379273 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.379274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:09.379922 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:09.379924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.380342 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.380343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:09.380412 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:09.380838 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:09.380843 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:09.382426 (+ 1583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88}
W20260924 03:03:09.382553 9283 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:09.378211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41211 (local address 127.0.98.62:44821)
0924 03:03:09.378259 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:09.378260 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:09.378298 (+ 38us) server_negotiation.cc:407] Connection header received
0924 03:03:09.378366 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:09.378367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:09.378376 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:09.378430 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:09.378664 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.379193 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.379981 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.380261 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.380329 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:09.382500 (+ 2171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41211: BlockingRecv error: recv got EOF from 127.0.98.3:41211 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 24 03:03:09 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:09 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:09.386435 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:09.382742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.3:49451)
0924 03:03:09.382842 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:09.382851 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:09.382889 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:09.382979 (+ 90us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:09.382980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:09.383080 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:09.383104 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:09.383105 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.383751 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.383752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:09.384218 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:09.384219 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.384522 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.384522 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:09.384556 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:09.384893 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:09.384897 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:09.386389 (+ 1492us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":82}
W20260924 03:03:09.386518 9283 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:09.382798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49451 (local address 127.0.98.62:44821)
0924 03:03:09.382848 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:09.382850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:09.382877 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:03:09.382924 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:09.382924 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:09.382931 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:09.382977 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:09.383152 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.383672 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.384298 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:09.384464 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:09.384516 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:09.386461 (+ 1945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49451: BlockingRecv error: recv got EOF from 127.0.98.3:49451 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260924 03:03:09.386605 9545 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 03:03:10 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:10 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:10.127760 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:10.122648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.1:37735)
0924 03:03:10.122796 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:10.122808 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:10.122867 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:10.123034 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:10.123036 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:10.123217 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:10.123268 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:10.123270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:10.124061 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:10.124063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:10.124683 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:10.124686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:10.125189 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:10.125190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:10.125256 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:10.125738 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:10.125743 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:10.127661 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
W20260924 03:03:10.127780 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:10.122757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37735 (local address 127.0.98.62:44821)
0924 03:03:10.122876 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:10.122877 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:10.122887 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:03:10.122920 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:10.122921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:10.122930 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:10.122989 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:10.123344 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:10.123978 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:10.124783 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:10.125096 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:10.125174 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:10.127712 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37735: BlockingRecv error: recv got EOF from 127.0.98.1:37735 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":67,"threads_started":1}
Sep 24 03:03:10 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:10 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:10.260564 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:10.255170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.2:43827)
0924 03:03:10.255350 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:10.255364 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:10.255442 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:10.255704 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:10.255705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:10.255893 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:10.255969 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:10.255972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:10.256789 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:10.256790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:10.257462 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:10.257464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:10.257895 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:10.257896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:10.257944 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:10.258533 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:10.258538 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:10.260494 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":107,"threads_started":1}
W20260924 03:03:10.260848 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:10.255298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43827 (local address 127.0.98.62:44821)
0924 03:03:10.255347 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:10.255350 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:10.255447 (+ 97us) server_negotiation.cc:407] Connection header received
0924 03:03:10.255540 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:10.255541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:10.255557 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:10.255690 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:10.256127 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:10.256668 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:10.257570 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:10.257812 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:10.257884 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:10.260745 (+ 2861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43827: BlockingRecv error: recv got EOF from 127.0.98.2:43827 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 24 03:03:10 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:10 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:10.391973 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:10.387033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.3:49233)
0924 03:03:10.387183 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:10.387194 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:10.387262 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:10.387410 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:10.387411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:10.387584 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:10.387655 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:10.387657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:10.388368 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:10.388369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:10.388977 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:10.388979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:10.389355 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:10.389356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:10.389408 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:10.389976 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:10.389982 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:10.391893 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":68,"threads_started":1}
W20260924 03:03:10.392108 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:10.387080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49233 (local address 127.0.98.62:44821)
0924 03:03:10.387137 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:10.387139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:10.387274 (+ 135us) server_negotiation.cc:407] Connection header received
0924 03:03:10.387301 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:10.387301 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:10.387313 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:10.387364 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:10.387751 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:10.388290 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:10.389106 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:10.389286 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:10.389344 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:10.392042 (+ 2698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49233: BlockingRecv error: recv got EOF from 127.0.98.3:49233 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 24 03:03:11 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:11 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:11.133129 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:11.128288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.1:36663)
0924 03:03:11.128438 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:11.128449 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:11.128511 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:11.128664 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:11.128665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:11.128827 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:11.128874 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:11.128876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:11.129647 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:11.129648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:11.130206 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:11.130208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:11.130567 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:11.130568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:11.130607 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:11.131180 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:11.131186 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:11.133034 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
W20260924 03:03:11.133183 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:11.128347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36663 (local address 127.0.98.62:44821)
0924 03:03:11.128472 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:11.128474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:11.128487 (+ 13us) server_negotiation.cc:407] Connection header received
0924 03:03:11.128567 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:11.128568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:11.128578 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:11.128641 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:11.128966 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:11.129568 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:11.130297 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:11.130498 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:11.130557 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:11.133110 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36663: BlockingRecv error: recv got EOF from 127.0.98.1:36663 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":77,"threads_started":1}
Sep 24 03:03:11 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:11 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:11.266162 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:11.261098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.2:33379)
0924 03:03:11.261352 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:11.261366 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:11.261445 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:11.261654 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:11.261655 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:11.261817 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:11.261865 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:11.261866 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:11.262561 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:11.262562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:11.263270 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:11.263272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:11.263657 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:11.263658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:11.263711 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:11.264155 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:11.264160 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:11.266098 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":78,"threads_started":1}
W20260924 03:03:11.266428 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:11.261229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33379 (local address 127.0.98.62:44821)
0924 03:03:11.261331 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:11.261332 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:11.261489 (+ 157us) server_negotiation.cc:407] Connection header received
0924 03:03:11.261523 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:11.261524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:11.261536 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:11.261585 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:11.261935 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:11.262460 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:11.263383 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:11.263569 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:11.263653 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:11.266218 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33379: BlockingRecv error: recv got EOF from 127.0.98.2:33379 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 24 03:03:11 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:11 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:11.397487 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:11.392467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.3:55305)
0924 03:03:11.392614 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:11.392632 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:11.392680 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:11.392828 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:11.392829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:11.392995 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:11.393046 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:11.393048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:11.393838 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:11.393839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:11.394409 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:11.394411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:11.394818 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:11.394819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:11.394889 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:11.395464 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:11.395469 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:11.397373 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
W20260924 03:03:11.397527 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:11.392620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55305 (local address 127.0.98.62:44821)
0924 03:03:11.392683 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:11.392685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:11.392730 (+ 45us) server_negotiation.cc:407] Connection header received
0924 03:03:11.392754 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:11.392755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:11.392765 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:11.392808 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:11.393151 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:11.393754 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:11.394514 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:11.394722 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:11.394815 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:11.397456 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55305: BlockingRecv error: recv got EOF from 127.0.98.3:55305 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 03:03:12 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:12 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:12.138454 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:12.133679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.1:39331)
0924 03:03:12.133796 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:12.133807 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:12.133865 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:12.134045 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:12.134046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:12.134201 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:12.134250 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:12.134252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:12.134948 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:12.134949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:12.135520 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:12.135523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:12.135972 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:12.135973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:12.136035 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:12.136546 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:12.136551 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:12.138381 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":68,"threads_started":1}
W20260924 03:03:12.138695 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:12.133747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39331 (local address 127.0.98.62:44821)
0924 03:03:12.133894 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:12.133896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:12.133906 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:03:12.133934 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:12.133935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:12.133947 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:12.134025 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:12.134311 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:12.134867 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:12.135662 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:12.135899 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:12.135958 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:12.138630 (+ 2672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39331: BlockingRecv error: recv got EOF from 127.0.98.1:39331 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
Sep 24 03:03:12 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:12 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:12.271746 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:12.266906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.2:34729)
0924 03:03:12.267062 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:12.267072 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:12.267130 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:12.267365 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:12.267366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:12.267604 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:12.267740 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:12.267748 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:12.268410 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:12.268411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:12.268936 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:12.268938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:12.269274 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:12.269275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:12.269317 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:12.269854 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:12.269858 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:12.271676 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
W20260924 03:03:12.271814 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:12.267014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34729 (local address 127.0.98.62:44821)
0924 03:03:12.267066 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:12.267068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:12.267117 (+ 49us) server_negotiation.cc:407] Connection header received
0924 03:03:12.267195 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:12.267196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:12.267206 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:12.267253 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:12.267825 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:12.268322 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:12.269012 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:12.269203 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:12.269261 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:12.271768 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34729: BlockingRecv error: recv got EOF from 127.0.98.2:34729 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 24 03:03:12 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:12 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:12.402977 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:12.398177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.3:49493)
0924 03:03:12.398345 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:12.398360 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:12.398439 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:12.398579 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:12.398580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:12.398758 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:12.398806 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:12.398808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:12.399522 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:12.399523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:12.400178 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:12.400181 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:12.400614 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:12.400614 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:12.400658 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:12.401088 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:12.401093 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:12.402918 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":105,"threads_started":1}
W20260924 03:03:12.403085 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:12.398367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49493 (local address 127.0.98.62:44821)
0924 03:03:12.398415 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:12.398417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:12.398429 (+ 12us) server_negotiation.cc:407] Connection header received
0924 03:03:12.398500 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:12.398501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:12.398514 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:12.398568 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:12.398913 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:12.399426 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:12.400257 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:12.400542 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:12.400595 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:12.403011 (+ 2416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49493: BlockingRecv error: recv got EOF from 127.0.98.3:49493 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 24 03:03:13 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:13 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:13.155911 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:13.139040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.1:43519)
0924 03:03:13.139173 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:13.139183 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:13.139232 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:13.139370 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:13.139370 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:13.139519 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:13.139568 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:13.139569 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:13.140474 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:13.140475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:13.141093 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:13.141095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:13.141468 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:13.141469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:13.141512 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:13.153808 (+ 12296us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:13.153813 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:13.155815 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
W20260924 03:03:13.155984 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:13.139115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43519 (local address 127.0.98.62:44821)
0924 03:03:13.139233 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:13.139245 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:13.139253 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:03:13.139276 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:13.139277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:13.139286 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:13.139356 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:13.139688 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:13.140397 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:13.141151 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:13.141398 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:13.141454 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:13.155913 (+ 14459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43519: BlockingRecv error: recv got EOF from 127.0.98.1:43519 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
Sep 24 03:03:13 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:13 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:13.277146 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:13.272310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.2:49529)
0924 03:03:13.272470 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:13.272484 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:13.272562 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:13.272764 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:13.272765 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:13.272931 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:13.272981 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:13.272983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:13.273717 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:13.273718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:13.274343 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:13.274345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:13.274807 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:13.274808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:13.274849 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:13.275371 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:13.275376 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:13.277089 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":70,"threads_started":1}
W20260924 03:03:13.277309 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:13.272374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49529 (local address 127.0.98.62:44821)
0924 03:03:13.272460 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:13.272462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:13.272548 (+ 86us) server_negotiation.cc:407] Connection header received
0924 03:03:13.272650 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:13.272651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:13.272661 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:13.272711 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:13.273049 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:13.273647 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:13.274437 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:13.274711 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:13.274804 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:13.277208 (+ 2404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49529: BlockingRecv error: recv got EOF from 127.0.98.2:49529 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 03:03:13 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:13 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:13.408440 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:13.403458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.3:35511)
0924 03:03:13.403700 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:13.403716 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:13.403800 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:13.403962 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:13.403962 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:13.404196 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:13.404248 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:13.404251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:13.404933 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:13.404933 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:13.405581 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:13.405584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:13.405953 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:13.405954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:13.406001 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:13.406525 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:13.406530 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:13.408379 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":148,"threads_started":1}
W20260924 03:03:13.408561 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:13.403506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35511 (local address 127.0.98.62:44821)
0924 03:03:13.403572 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:13.403575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:13.403793 (+ 218us) server_negotiation.cc:407] Connection header received
0924 03:03:13.403874 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:13.403875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:13.403888 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:13.403941 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:13.404325 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:13.404853 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:13.405675 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:13.405869 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:13.405930 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:13.408477 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35511: BlockingRecv error: recv got EOF from 127.0.98.3:35511 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 03:03:14 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:14 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:14.161669 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:14.156433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.1:45275)
0924 03:03:14.156591 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:14.156603 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:14.156675 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:14.156848 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:14.156849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:14.157034 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:14.157089 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:14.157091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:14.157846 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:14.157847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:14.158500 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:14.158502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:14.158856 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:14.158857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:14.158910 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:14.159493 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:14.159499 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:14.161599 (+ 2100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
W20260924 03:03:14.161789 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:14.156524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45275 (local address 127.0.98.62:44821)
0924 03:03:14.156624 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:14.156626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:14.156655 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:03:14.156729 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:14.156730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:14.156740 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:14.156793 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:14.157244 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:14.157759 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:14.158579 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:14.158761 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:14.158815 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:14.161685 (+ 2870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45275: BlockingRecv error: recv got EOF from 127.0.98.1:45275 (error 108)
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":48,"threads_started":1}
Sep 24 03:03:14 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:14 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:14.282724 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:14.277691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.2:59973)
0924 03:03:14.277829 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:14.277841 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:14.277898 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:14.278064 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:14.278065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:14.278310 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:14.278357 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:14.278360 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:14.279079 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:14.279080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:14.279930 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:14.279932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:14.280306 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:14.280306 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:14.280349 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:14.280838 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:14.280843 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:14.282656 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
W20260924 03:03:14.282845 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:14.277837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59973 (local address 127.0.98.62:44821)
0924 03:03:14.277927 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:14.277929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:14.277938 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:03:14.277975 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:14.277975 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:14.277985 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:14.278046 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:14.278431 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:14.279010 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:14.279999 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:14.280197 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:14.280256 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:14.282745 (+ 2489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59973: BlockingRecv error: recv got EOF from 127.0.98.2:59973 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 03:03:14 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:14 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:14.413873 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:14.408996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.3:48633)
0924 03:03:14.409139 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:14.409157 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:14.409227 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:14.409372 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:14.409373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:14.409528 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:14.409575 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:14.409577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:14.410316 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:14.410317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:14.411009 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:14.411011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:14.411352 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:14.411353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:14.411393 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:14.411829 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:14.411833 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:14.413748 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
W20260924 03:03:14.413930 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:14.409110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48633 (local address 127.0.98.62:44821)
0924 03:03:14.409200 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:14.409202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:14.409213 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:03:14.409293 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:14.409293 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:14.409305 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:14.409348 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:14.409635 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:14.410241 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:14.411082 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:14.411286 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:14.411343 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:14.413865 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48633: BlockingRecv error: recv got EOF from 127.0.98.3:48633 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 03:03:15 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:15 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:15.167389 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:15.162238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.1:39515)
0924 03:03:15.162399 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:15.162409 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:15.162463 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:15.162715 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:15.162716 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:15.162898 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:15.162951 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:15.162953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:15.163713 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:15.163714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:15.164384 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:15.164386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:15.164788 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:15.164789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:15.164847 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:15.165366 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:15.165371 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:15.167317 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":71,"threads_started":1}
W20260924 03:03:15.167577 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:15.162402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39515 (local address 127.0.98.62:44821)
0924 03:03:15.162537 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:15.162539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:15.162549 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:03:15.162577 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:15.162577 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:15.162589 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:15.162667 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:15.163025 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:15.163567 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:15.164455 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:15.164726 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:15.164801 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:15.167489 (+ 2688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39515: BlockingRecv error: recv got EOF from 127.0.98.1:39515 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":73,"threads_started":1}
Sep 24 03:03:15 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:15 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:15.288007 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:15.283223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.2:35731)
0924 03:03:15.283443 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:15.283456 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:15.283551 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:15.283703 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:15.283704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:15.283899 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:15.283949 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:15.283951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:15.284597 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:15.284598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:15.285144 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:15.285145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:15.285502 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:15.285503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:15.285557 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:15.286004 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:15.286008 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:15.287946 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":145,"threads_started":1}
W20260924 03:03:15.288148 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:15.283343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35731 (local address 127.0.98.62:44821)
0924 03:03:15.283445 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:15.283447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:15.283531 (+ 84us) server_negotiation.cc:407] Connection header received
0924 03:03:15.283630 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:15.283631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:15.283644 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:15.283699 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:15.284028 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:15.284520 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:15.285227 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:15.285419 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:15.285501 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:15.288062 (+ 2561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35731: BlockingRecv error: recv got EOF from 127.0.98.2:35731 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 03:03:15 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:15 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:15.419749 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:15.414443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.3:39957)
0924 03:03:15.414567 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:15.414577 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:15.414635 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:15.414763 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:15.414764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:15.414920 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:15.414969 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:15.414970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:15.415732 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:15.415733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:15.416403 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:15.416406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:15.416940 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:15.416942 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:15.417011 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:15.417584 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:15.417589 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:15.419680 (+ 2091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":67,"threads_started":1}
W20260924 03:03:15.419875 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:15.414513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39957 (local address 127.0.98.62:44821)
0924 03:03:15.414582 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:15.414585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:15.414633 (+ 48us) server_negotiation.cc:407] Connection header received
0924 03:03:15.414680 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:15.414681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:15.414693 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:15.414758 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:15.415066 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:15.415657 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:15.416536 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:15.416844 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:15.416915 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:15.419794 (+ 2879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39957: BlockingRecv error: recv got EOF from 127.0.98.3:39957 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 03:03:16 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:16 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:16.172716 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:16.167902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.1:41339)
0924 03:03:16.168025 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:16.168038 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:16.168113 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:16.168252 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:16.168252 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:16.168430 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:16.168481 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:16.168483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:16.169141 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:16.169142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:16.169677 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:16.169679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:16.169999 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:16.170000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:16.170039 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:16.170750 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:16.170756 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:16.172652 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
W20260924 03:03:16.172854 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:16.167986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41339 (local address 127.0.98.62:44821)
0924 03:03:16.168111 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:16.168113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:16.168123 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:03:16.168166 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:16.168167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:16.168179 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:16.168238 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:16.168543 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:16.169067 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:16.169756 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:16.169937 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:16.169988 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:16.172765 (+ 2777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41339: BlockingRecv error: recv got EOF from 127.0.98.1:41339 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
Sep 24 03:03:16 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:16 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:16.293316 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:16.288651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.2:43471)
0924 03:03:16.288853 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:16.288865 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:16.288950 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:16.289112 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:16.289113 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:16.289294 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:16.289344 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:16.289346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:16.290013 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:16.290014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:16.290608 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:16.290610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:16.290960 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:16.290961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:16.291015 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:16.291454 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:16.291458 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:16.293218 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":124,"threads_started":1}
W20260924 03:03:16.293423 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:16.288761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43471 (local address 127.0.98.62:44821)
0924 03:03:16.288832 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:16.288835 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:16.288931 (+ 96us) server_negotiation.cc:407] Connection header received
0924 03:03:16.289019 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:16.289020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:16.289036 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:16.289091 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:16.289436 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:16.289934 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:16.290688 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:16.290878 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:16.290958 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:16.293304 (+ 2346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43471: BlockingRecv error: recv got EOF from 127.0.98.2:43471 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 03:03:16 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:16 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:16.425588 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:16.420358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.3:52003)
0924 03:03:16.420582 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:16.420595 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:16.420690 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:16.420883 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:16.420884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:16.421074 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:16.421126 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:16.421128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:16.421911 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:16.421912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:16.422723 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:16.422730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:16.423125 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:16.423126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:16.423190 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:16.423683 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:16.423697 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:16.425529 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":114,"threads_started":1}
W20260924 03:03:16.425736 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:16.420508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52003 (local address 127.0.98.62:44821)
0924 03:03:16.420581 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:16.420583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:16.420668 (+ 85us) server_negotiation.cc:407] Connection header received
0924 03:03:16.420784 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:16.420785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:16.420805 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:16.420875 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:16.421208 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:16.421782 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:16.422807 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:16.423024 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:16.423114 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:16.425643 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52003: BlockingRecv error: recv got EOF from 127.0.98.3:52003 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 03:03:17 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:17 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:17.178120 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:17.173210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.1:48677)
0924 03:03:17.173388 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:17.173402 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:17.173474 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:17.173780 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:17.173781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:17.173936 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:17.173983 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:17.173985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:17.174627 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:17.174628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:17.175228 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:17.175230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:17.175698 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:17.175700 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:17.175753 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:17.176286 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:17.176293 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:17.178062 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":85,"threads_started":1}
W20260924 03:03:17.178215 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:17.173439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48677 (local address 127.0.98.62:44821)
0924 03:03:17.173541 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:17.173543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:17.173552 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:03:17.173579 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:17.173580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:17.173590 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:17.173686 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:17.174053 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:17.174546 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:17.175297 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:17.175528 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:17.175584 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:17.178142 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48677: BlockingRecv error: recv got EOF from 127.0.98.1:48677 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":58,"threads_started":1}
Sep 24 03:03:17 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:17 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:17.298750 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:17.293817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.2:37827)
0924 03:03:17.293991 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:17.294003 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:17.294062 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:17.294213 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:17.294213 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:17.294392 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:17.294442 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:17.294444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:17.295189 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:17.295190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:17.295853 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:17.295856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:17.296304 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:17.296305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:17.296370 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:17.296857 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:17.296863 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:17.298690 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":110,"threads_started":1}
W20260924 03:03:17.298821 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:17.293937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37827 (local address 127.0.98.62:44821)
0924 03:03:17.294000 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:17.294002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:17.294047 (+ 45us) server_negotiation.cc:407] Connection header received
0924 03:03:17.294118 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:17.294119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:17.294129 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:17.294194 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:17.294508 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:17.295122 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:17.295968 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:17.296217 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:17.296306 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:17.298775 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37827: BlockingRecv error: recv got EOF from 127.0.98.2:37827 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 03:03:17 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:17 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:17.431296 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:17.426246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.3:40717)
0924 03:03:17.426384 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:17.426395 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:17.426461 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:17.426684 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:17.426685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:17.426907 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:17.426970 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:17.426972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:17.427791 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:17.427792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:17.428469 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:17.428471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:17.428831 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:17.428832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:17.428874 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:17.429428 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:17.429434 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:17.431233 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
W20260924 03:03:17.431493 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:17.426319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40717 (local address 127.0.98.62:44821)
0924 03:03:17.426387 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:17.426389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:17.426464 (+ 75us) server_negotiation.cc:407] Connection header received
0924 03:03:17.426552 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:17.426553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:17.426580 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:17.426655 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:17.427046 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:17.427705 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:17.428551 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:17.428743 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:17.428824 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:17.431398 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40717: BlockingRecv error: recv got EOF from 127.0.98.3:40717 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 03:03:18 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:18 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:18.189816 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:18.178635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.1:60167)
0924 03:03:18.178810 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:18.178825 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:18.178900 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:18.179076 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:18.179077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:18.179278 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:18.179330 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:18.179332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:18.180096 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:18.180097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:18.180628 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:18.180630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:18.180999 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:18.181000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:18.181042 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:18.187843 (+ 6801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:18.187849 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:18.189757 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":139,"threads_started":1}
W20260924 03:03:18.190013 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:18.178710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60167 (local address 127.0.98.62:44821)
0924 03:03:18.178868 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:18.178870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:18.178878 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:03:18.178965 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:18.178966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:18.178976 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:18.179037 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:18.179408 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:18.180020 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:18.180715 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:18.180931 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:18.180989 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:18.189942 (+ 8953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60167: BlockingRecv error: recv got EOF from 127.0.98.1:60167 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
Sep 24 03:03:18 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:18 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:18.304044 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:18.299279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.2:53585)
0924 03:03:18.299416 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:18.299426 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:18.299493 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:18.299659 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:18.299659 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:18.299820 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:18.299869 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:18.299870 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:18.300590 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:18.300591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:18.301128 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:18.301130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:18.301523 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:18.301524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:18.301565 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:18.302052 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:18.302057 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:18.303981 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
W20260924 03:03:18.304165 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:18.299337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53585 (local address 127.0.98.62:44821)
0924 03:03:18.299431 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:18.299433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:18.299477 (+ 44us) server_negotiation.cc:407] Connection header received
0924 03:03:18.299548 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:18.299549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:18.299559 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:18.299648 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:18.299933 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:18.300510 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:18.301205 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:18.301456 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:18.301516 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:18.304068 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53585: BlockingRecv error: recv got EOF from 127.0.98.2:53585 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 03:03:18 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:18 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:18.436374 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:18.431765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.3:36065)
0924 03:03:18.431893 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:18.431905 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:18.431971 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:18.432138 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:18.432138 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:18.432300 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:18.432393 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:18.432395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:18.433027 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:18.433027 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:18.433580 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:18.433582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:18.433931 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:18.433931 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:18.433972 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:18.434436 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:18.434440 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:18.436315 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
W20260924 03:03:18.436518 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:18.431862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36065 (local address 127.0.98.62:44821)
0924 03:03:18.431923 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:18.431925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:18.431956 (+ 31us) server_negotiation.cc:407] Connection header received
0924 03:03:18.432051 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:18.432052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:18.432065 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:18.432130 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:18.432461 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:18.432964 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:18.433659 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:18.433866 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:18.433920 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:18.436432 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36065: BlockingRecv error: recv got EOF from 127.0.98.3:36065 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 03:03:19 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:19 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:19.195787 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:19.190448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.1:37871)
0924 03:03:19.190582 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:19.190595 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:19.190655 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:19.190816 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:19.190817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:19.190988 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:19.191036 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:19.191038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:19.191866 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:19.191868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:19.192450 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:19.192453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:19.192852 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:19.192853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:19.192896 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:19.193527 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:19.193532 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:19.195724 (+ 2192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
W20260924 03:03:19.195917 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:19.190532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37871 (local address 127.0.98.62:44821)
0924 03:03:19.190690 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:19.190692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:19.190699 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:03:19.190720 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:19.190721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:19.190732 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:19.190796 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:19.191139 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:19.191712 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:19.192551 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:19.192764 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:19.192835 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:19.195827 (+ 2992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37871: BlockingRecv error: recv got EOF from 127.0.98.1:37871 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":103,"threads_started":1}
Sep 24 03:03:19 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:19 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:19.309300 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:19.304583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.2:42919)
0924 03:03:19.304745 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:19.304755 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:19.304818 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:19.304941 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:19.304941 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:19.305101 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:19.305148 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:19.305149 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:19.305802 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:19.305803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:19.306358 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:19.306360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:19.306695 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:19.306695 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:19.306735 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:19.307273 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:19.307278 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:19.309233 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":109,"threads_started":1}
W20260924 03:03:19.309499 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:19.304639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42919 (local address 127.0.98.62:44821)
0924 03:03:19.304705 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:19.304707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:19.304823 (+ 116us) server_negotiation.cc:407] Connection header received
0924 03:03:19.304876 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:19.304877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:19.304887 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:19.304936 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:19.305225 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:19.305723 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:19.306436 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:19.306626 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:19.306683 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:19.309414 (+ 2731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42919: BlockingRecv error: recv got EOF from 127.0.98.2:42919 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 03:03:19 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:19 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:19.442272 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:19.436891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.3:60507)
0924 03:03:19.437068 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:19.437082 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:19.437151 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:19.437393 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:19.437394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:19.437593 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:19.437643 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:19.437645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:19.438391 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:19.438392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:19.438993 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:19.438995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:19.439415 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:19.439416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:19.439485 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:19.440077 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:19.440082 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:19.442181 (+ 2099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":116,"threads_started":1}
W20260924 03:03:19.442420 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:19.437057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60507 (local address 127.0.98.62:44821)
0924 03:03:19.437122 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:19.437124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:19.437134 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:03:19.437215 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:19.437216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:19.437231 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:19.437383 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:19.437736 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:19.438296 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:19.439091 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:19.439309 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:19.439414 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:19.442298 (+ 2884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60507: BlockingRecv error: recv got EOF from 127.0.98.3:60507 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 24 03:03:20 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:20 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:20.201092 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:20.196349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.1:56639)
0924 03:03:20.196531 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:20.196542 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:20.196621 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:20.196786 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:20.196786 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:20.196975 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:20.197021 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:20.197022 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:20.197705 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:20.197706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:20.198239 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:20.198240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:20.198627 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:20.198628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:20.198683 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:20.199150 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:20.199155 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:20.201030 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":103,"threads_started":1}
W20260924 03:03:20.201251 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:20.196354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56639 (local address 127.0.98.62:44821)
0924 03:03:20.196510 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:20.196513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:20.196605 (+ 92us) server_negotiation.cc:407] Connection header received
0924 03:03:20.196675 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:20.196676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:20.196689 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:20.196763 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:20.197112 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:20.197624 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:20.198325 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:20.198538 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:20.198625 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:20.201154 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56639: BlockingRecv error: recv got EOF from 127.0.98.1:56639 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":79,"threads_started":1}
Sep 24 03:03:20 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:20 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:20.314766 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:20.309807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.2:37559)
0924 03:03:20.309978 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:20.309988 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:20.310045 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:20.310169 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:20.310169 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:20.310328 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:20.310377 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:20.310378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:20.311115 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:20.311116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:20.311686 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:20.311688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:20.312028 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:20.312029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:20.312084 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:20.312508 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:20.312512 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:20.314652 (+ 2140us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":108,"threads_started":1}
W20260924 03:03:20.314795 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:20.309902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37559 (local address 127.0.98.62:44821)
0924 03:03:20.309952 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:20.309954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:20.310040 (+ 86us) server_negotiation.cc:407] Connection header received
0924 03:03:20.310108 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:20.310109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:20.310120 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:20.310156 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:20.310541 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:20.311036 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:20.311754 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:20.311938 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:20.311995 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:20.314735 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37559: BlockingRecv error: recv got EOF from 127.0.98.2:37559 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 24 03:03:20 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:20 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:20.448036 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:20.442839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.3:60763)
0924 03:03:20.443023 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:20.443036 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:20.443148 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:20.443353 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:20.443354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:20.443546 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:20.443632 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:20.443634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:20.444344 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:20.444346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:20.444888 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:20.444890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:20.445259 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:20.445260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:20.445333 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:20.445847 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:20.445857 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:20.447930 (+ 2073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":94,"threads_started":1}
W20260924 03:03:20.448086 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:20.442938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60763 (local address 127.0.98.62:44821)
0924 03:03:20.443025 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:20.443027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:20.443126 (+ 99us) server_negotiation.cc:407] Connection header received
0924 03:03:20.443218 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:20.443219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:20.443240 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:20.443333 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:20.443724 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:20.444239 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:20.444978 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:20.445173 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:20.445258 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:20.448016 (+ 2758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60763: BlockingRecv error: recv got EOF from 127.0.98.3:60763 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 03:03:21 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:21 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:21.206449 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:21.201612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.1:49817)
0924 03:03:21.201787 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:21.201801 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:21.201871 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:21.202055 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:21.202056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:21.202219 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:21.202271 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:21.202273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:21.203008 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:21.203009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:21.203576 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:21.203578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:21.203980 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:21.203980 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:21.204018 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:21.204522 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:21.204526 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:21.206381 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":80,"threads_started":1}
W20260924 03:03:21.206476 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:21.201694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49817 (local address 127.0.98.62:44821)
0924 03:03:21.201860 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:21.201862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:21.201871 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:03:21.201929 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:21.201929 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:21.201940 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:21.202002 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:21.202346 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:21.202935 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:21.203689 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:21.203911 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:21.203971 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:21.206420 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49817: BlockingRecv error: recv got EOF from 127.0.98.1:49817 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
Sep 24 03:03:21 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:21 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:21.320269 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:21.315434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.2:51175)
0924 03:03:21.315574 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:21.315612 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:21.315675 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:21.315848 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:21.315848 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:21.316038 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:21.316090 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:21.316091 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:21.316822 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:21.316823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:21.317431 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:21.317434 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:21.317774 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:21.317775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:21.317826 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:21.318259 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:21.318264 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:21.320200 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
W20260924 03:03:21.320405 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:21.315538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51175 (local address 127.0.98.62:44821)
0924 03:03:21.315620 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:21.315622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:21.315663 (+ 41us) server_negotiation.cc:407] Connection header received
0924 03:03:21.315753 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:21.315754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:21.315764 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:21.315820 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:21.316173 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:21.316723 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:21.317512 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:21.317690 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:21.317742 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:21.320323 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51175: BlockingRecv error: recv got EOF from 127.0.98.2:51175 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 03:03:21 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:21 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:21.453977 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:21.448551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.3:59945)
0924 03:03:21.448720 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:21.448733 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:21.448797 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:21.449053 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:21.449054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:21.449238 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:21.449288 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:21.449290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:21.450099 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:21.450100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:21.450732 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:21.450734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:21.451106 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:21.451107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:21.451149 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:21.451757 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:21.451767 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:21.453902 (+ 2135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":118,"threads_started":1}
W20260924 03:03:21.454105 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:21.448698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59945 (local address 127.0.98.62:44821)
0924 03:03:21.448761 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:21.448763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:21.448778 (+ 15us) server_negotiation.cc:407] Connection header received
0924 03:03:21.448904 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:21.448905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:21.448917 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:21.448993 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:21.449469 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:21.450012 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:21.450821 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:21.451029 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:21.451090 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:21.453992 (+ 2902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59945: BlockingRecv error: recv got EOF from 127.0.98.3:59945 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 03:03:22 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:22 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:22.211899 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:22.206929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.1:46059)
0924 03:03:22.207072 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:22.207083 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:22.207123 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:22.207266 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:22.207267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:22.207428 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:22.207481 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:22.207483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:22.208213 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:22.208215 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:22.208875 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:22.208878 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:22.209283 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:22.209284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:22.209328 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:22.209836 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:22.209841 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:22.211792 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
W20260924 03:03:22.211970 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:22.206975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46059 (local address 127.0.98.62:44821)
0924 03:03:22.207102 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:22.207104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:22.207127 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:03:22.207200 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:22.207201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:22.207211 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:22.207263 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:22.207539 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:22.208119 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:22.208989 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:22.209203 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:22.209271 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:22.211889 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46059: BlockingRecv error: recv got EOF from 127.0.98.1:46059 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
Sep 24 03:03:22 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:22 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:22.325951 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:22.320874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.2:42803)
0924 03:03:22.321041 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:22.321052 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:22.321101 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:22.321294 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:22.321295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:22.321484 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:22.321530 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:22.321532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:22.322226 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:22.322226 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:22.322811 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:22.322813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:22.323163 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:22.323164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:22.323207 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:22.323860 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:22.323866 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:22.325843 (+ 1977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":103,"threads_started":1}
W20260924 03:03:22.326016 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:22.321025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42803 (local address 127.0.98.62:44821)
0924 03:03:22.321096 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:22.321099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:22.321108 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:03:22.321162 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:22.321163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:22.321174 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:22.321230 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:22.321639 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:22.322142 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:22.322896 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:22.323091 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:22.323150 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:22.325927 (+ 2777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42803: BlockingRecv error: recv got EOF from 127.0.98.2:42803 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 03:03:22 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:22 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:22.459372 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:22.454476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.3:33891)
0924 03:03:22.454666 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:22.454682 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:22.454757 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:22.454901 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:22.454902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:22.455081 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:22.455131 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:22.455132 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:22.455828 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:22.455829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:22.456698 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:22.456701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:22.457004 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:22.457005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:22.457043 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:22.457547 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:22.457553 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:22.459313 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":73,"threads_started":1}
W20260924 03:03:22.459548 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:22.454640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33891 (local address 127.0.98.62:44821)
0924 03:03:22.454705 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:22.454706 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:22.454740 (+ 34us) server_negotiation.cc:407] Connection header received
0924 03:03:22.454812 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:22.454813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:22.454825 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:22.454875 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:22.455198 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:22.455758 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:22.456763 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:22.456941 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:22.456995 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:22.459431 (+ 2436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33891: BlockingRecv error: recv got EOF from 127.0.98.3:33891 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 03:03:23 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:23 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 03:03:23.226516 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:23.212509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58199 (local address 127.0.98.62:44821)
0924 03:03:23.212648 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:23.212650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:23.212684 (+ 34us) server_negotiation.cc:407] Connection header received
0924 03:03:23.212755 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:23.212756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:23.212771 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:23.212860 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:23.213199 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:23.213941 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:23.214893 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:23.215151 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:23.215223 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:23.226455 (+ 11232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58199: BlockingRecv error: recv got EOF from 127.0.98.1:58199 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260924 03:03:23.226536 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:23.212435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.1:58199)
0924 03:03:23.212596 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:23.212616 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:23.212697 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:23.212873 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:23.212874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:23.213068 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:23.213121 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:23.213123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:23.214037 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:23.214038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:23.214781 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:23.214784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:23.215236 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:23.215237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:23.215280 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:23.224187 (+ 8907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:23.224193 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:23.226359 (+ 2166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":105,"threads_started":1}
Sep 24 03:03:23 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:23 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:23.330888 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:23.326455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.2:49053)
0924 03:03:23.326640 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:23.326654 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:23.326720 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:23.326862 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:23.326862 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:23.327032 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:23.327078 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:23.327079 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:23.327790 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:23.327791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:23.328317 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:23.328318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:23.328692 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:23.328693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:23.328734 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:23.329161 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:23.329165 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:23.330830 (+ 1665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":86,"threads_started":1}
W20260924 03:03:23.331054 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:23.326627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49053 (local address 127.0.98.62:44821)
0924 03:03:23.326691 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:23.326692 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:23.326705 (+ 13us) server_negotiation.cc:407] Connection header received
0924 03:03:23.326780 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:23.326781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:23.326792 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:23.326840 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:23.327169 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:23.327712 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:23.328390 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:23.328618 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:23.328678 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:23.330946 (+ 2268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49053: BlockingRecv error: recv got EOF from 127.0.98.2:49053 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 03:03:23 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:23 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:23.465006 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:23.459837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.3:60589)
0924 03:03:23.460011 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:23.460025 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:23.460123 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:23.460330 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:23.460331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:23.460544 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:23.460593 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:23.460596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:23.461305 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:23.461306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:23.461845 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:23.461848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:23.462257 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:23.462259 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:23.462325 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:23.462907 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:23.462913 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:23.464903 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
W20260924 03:03:23.465056 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:23.459977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60589 (local address 127.0.98.62:44821)
0924 03:03:23.460045 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:23.460047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:23.460098 (+ 51us) server_negotiation.cc:407] Connection header received
0924 03:03:23.460215 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:23.460216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:23.460232 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:23.460307 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:23.460688 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:23.461199 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:23.461952 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:23.462148 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:23.462243 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:23.465007 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60589: BlockingRecv error: recv got EOF from 127.0.98.3:60589 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 03:03:24 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:24 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:24.232656 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:24.227228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.1:39487)
0924 03:03:24.227467 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:24.227481 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:24.227672 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:24.227900 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:24.227901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:24.228120 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:24.228171 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:24.228173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:24.228879 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:24.228880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:24.229544 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:24.229547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:24.230102 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:24.230103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:24.230165 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:24.230728 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:24.230733 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:24.232546 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":74,"threads_started":1}
W20260924 03:03:24.232709 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:24.227305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39487 (local address 127.0.98.62:44821)
0924 03:03:24.227425 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:24.227428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:24.227659 (+ 231us) server_negotiation.cc:407] Connection header received
0924 03:03:24.227784 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:24.227785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:24.227799 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:24.227874 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:24.228249 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:24.228781 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:24.229645 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:24.230005 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:24.230092 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:24.232635 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39487: BlockingRecv error: recv got EOF from 127.0.98.1:39487 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":68,"threads_started":1}
Sep 24 03:03:24 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:03:24 dist-test-slave-wgsb krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:03:24.336661 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:24.331469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44821 (local address 127.0.98.2:33575)
0924 03:03:24.331702 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:24.331718 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:24.331787 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:24.331924 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:24.331925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:24.332108 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:24.332158 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:24.332159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:24.332892 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:24.332893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:24.333486 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:24.333488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:24.333861 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:24.333862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:24.333914 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:24.334497 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:24.334503 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:24.336589 (+ 2086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44821: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":139,"threads_started":1}
W20260924 03:03:24.336828 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:24.331683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33575 (local address 127.0.98.62:44821)
0924 03:03:24.331764 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:24.331766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:24.331775 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:03:24.331840 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:24.331841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:24.331850 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:24.331902 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:24.332257 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:24.332810 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:24.333572 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:24.333766 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:24.333823 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:24.336738 (+ 2915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33575: BlockingRecv error: recv got EOF from 127.0.98.2:33575 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 03:03:24.379683 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 9132
I20260924 03:03:24.384518 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 9284
I20260924 03:03:24.389282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 9416
I20260924 03:03:24.393908 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 9062
2026-09-24T03:03:24Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17565 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T03:03:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:03:24Z Disabled control of system clock
I20260924 03:03:24.422636 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43639
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 03:03:24.504478 9612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:03:24.504691 9612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:03:24.504711 9612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:03:24.505851 9612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:03:24.505888 9612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:03:24.505900 9612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:03:24.505913 9612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:03:24.505925 9612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:03:24.507239 9612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43639
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:03:24.507426 9612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:03:24.507843 9612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:03:24.512929 9612 server_base.cc:1034] running on GCE node
W20260924 03:03:24.512993 9617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:24.513060 9620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:24.513010 9618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:24.513394 9612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:03:24.513700 9612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:03:24.514866 9612 hybrid_clock.cc:648] HybridClock initialized: now 1790219004514848 us; error 29 us; skew 500 ppm
I20260924 03:03:24.516186 9612 webserver.cc:460] Webserver started at http://127.0.98.62:35363/ using document root <none> and password file <none>
I20260924 03:03:24.516410 9612 fs_manager.cc:359] Metadata directory not provided
I20260924 03:03:24.516474 9612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:03:24.516637 9612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:03:24.517774 9612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "7df6bc9d107f4d0384920e0c6c6fb3f0"
format_stamp: "Formatted at 2026-09-24 03:03:24 on dist-test-slave-wgsb"
server_key: "5d85fb0d2f165a5757b61c6f0e7d11ef"
server_key_iv: "71e67ce6aa542da33583fa39984d70a8"
server_key_version: "encryptionkey@0"
I20260924 03:03:24.518078 9612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "7df6bc9d107f4d0384920e0c6c6fb3f0"
format_stamp: "Formatted at 2026-09-24 03:03:24 on dist-test-slave-wgsb"
server_key: "5d85fb0d2f165a5757b61c6f0e7d11ef"
server_key_iv: "71e67ce6aa542da33583fa39984d70a8"
server_key_version: "encryptionkey@0"
I20260924 03:03:24.519284 9612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 03:03:24.519964 9626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:24.520162 9612 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:24.520269 9612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "7df6bc9d107f4d0384920e0c6c6fb3f0"
format_stamp: "Formatted at 2026-09-24 03:03:24 on dist-test-slave-wgsb"
server_key: "5d85fb0d2f165a5757b61c6f0e7d11ef"
server_key_iv: "71e67ce6aa542da33583fa39984d70a8"
server_key_version: "encryptionkey@0"
I20260924 03:03:24.520344 9612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:03:24.553701 9612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:03:24.554054 9612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:03:24.554217 9612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:03:24.558404 9612 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43639
I20260924 03:03:24.558441 9678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43639 every 8 connection(s)
I20260924 03:03:24.558815 9612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:03:24.559489 9679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:03:24.562235 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 9612
I20260924 03:03:24.562294 9679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7df6bc9d107f4d0384920e0c6c6fb3f0: Bootstrap starting.
I20260924 03:03:24.562341 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:03:24.562542 392 external_mini_cluster.cc:1428] Setting key 77afd127053c707d7d9c364524573bc5
I20260924 03:03:24.563027 9679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7df6bc9d107f4d0384920e0c6c6fb3f0: Neither blocks nor log segments found. Creating new log.
I20260924 03:03:24.563426 9679 log.cc:824] T 00000000000000000000000000000000 P 7df6bc9d107f4d0384920e0c6c6fb3f0: Log is configured to *not* fsync() on all Append() calls
I20260924 03:03:24.564208 9679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7df6bc9d107f4d0384920e0c6c6fb3f0: No bootstrap required, opened a new log
I20260924 03:03:24.566345 9679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7df6bc9d107f4d0384920e0c6c6fb3f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df6bc9d107f4d0384920e0c6c6fb3f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43639 } }
I20260924 03:03:24.566531 9679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7df6bc9d107f4d0384920e0c6c6fb3f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:03:24.566576 9679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7df6bc9d107f4d0384920e0c6c6fb3f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df6bc9d107f4d0384920e0c6c6fb3f0, State: Initialized, Role: FOLLOWER
I20260924 03:03:24.566727 9679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7df6bc9d107f4d0384920e0c6c6fb3f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df6bc9d107f4d0384920e0c6c6fb3f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43639 } }
I20260924 03:03:24.566840 9679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7df6bc9d107f4d0384920e0c6c6fb3f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:03:24.566886 9679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7df6bc9d107f4d0384920e0c6c6fb3f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:03:24.566937 9679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7df6bc9d107f4d0384920e0c6c6fb3f0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:03:24.567709 9679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7df6bc9d107f4d0384920e0c6c6fb3f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df6bc9d107f4d0384920e0c6c6fb3f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43639 } }
I20260924 03:03:24.567821 9679 leader_election.cc:302] T 00000000000000000000000000000000 P 7df6bc9d107f4d0384920e0c6c6fb3f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7df6bc9d107f4d0384920e0c6c6fb3f0; no voters:
I20260924 03:03:24.567991 9679 leader_election.cc:288] T 00000000000000000000000000000000 P 7df6bc9d107f4d0384920e0c6c6fb3f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:03:24.568081 9684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7df6bc9d107f4d0384920e0c6c6fb3f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:03:24.568307 9679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7df6bc9d107f4d0384920e0c6c6fb3f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:03:24.568305 9684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7df6bc9d107f4d0384920e0c6c6fb3f0 [term 1 LEADER]: Becoming Leader. State: Replica: 7df6bc9d107f4d0384920e0c6c6fb3f0, State: Running, Role: LEADER
I20260924 03:03:24.568538 9684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7df6bc9d107f4d0384920e0c6c6fb3f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df6bc9d107f4d0384920e0c6c6fb3f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43639 } }
I20260924 03:03:24.568944 9685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7df6bc9d107f4d0384920e0c6c6fb3f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7df6bc9d107f4d0384920e0c6c6fb3f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df6bc9d107f4d0384920e0c6c6fb3f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43639 } } }
I20260924 03:03:24.568954 9686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7df6bc9d107f4d0384920e0c6c6fb3f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7df6bc9d107f4d0384920e0c6c6fb3f0. Latest consensus state: current_term: 1 leader_uuid: "7df6bc9d107f4d0384920e0c6c6fb3f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df6bc9d107f4d0384920e0c6c6fb3f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43639 } } }
I20260924 03:03:24.569129 9686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7df6bc9d107f4d0384920e0c6c6fb3f0 [sys.catalog]: This master's current role is: LEADER
I20260924 03:03:24.569206 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:24.563154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44054 (local address 127.0.98.62:43639)
0924 03:03:24.563390 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:24.563397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:24.563425 (+ 28us) server_negotiation.cc:407] Connection header received
0924 03:03:24.564092 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:24.564099 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:24.564159 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:24.564331 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:24.564550 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:24.565401 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:24.566577 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:24.567135 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:24.567230 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:24.568115 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:24.568119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:24.568120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:03:24.568332 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:24.568378 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:03:24.568394 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:24.568455 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:24.568741 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 03:03:24.568821 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":117,"threads_started":1}
I20260924 03:03:24.569664 9685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7df6bc9d107f4d0384920e0c6c6fb3f0 [sys.catalog]: This master's current role is: LEADER
W20260924 03:03:24.570206 9700 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7df6bc9d107f4d0384920e0c6c6fb3f0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:03:24.570273 9700 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:03:24.570329 9693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:03:24.570475 9693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:03:24.571985 9693 catalog_manager.cc:1347] Generated new cluster ID: a14c02fe91b34524bab278d4b83031ba
I20260924 03:03:24.572049 9693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:03:24.597944 9693 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:03:24.598593 9693 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:03:24.605118 9693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7df6bc9d107f4d0384920e0c6c6fb3f0: Generated new TSK 0
I20260924 03:03:24.605319 9693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:03:24.613430 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43639
--builtin_ntp_servers=127.0.98.20:38893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 03:03:24.696079 9703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:03:24.696295 9703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:03:24.696317 9703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:03:24.697417 9703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:03:24.697449 9703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:03:24.697474 9703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:03:24.698838 9703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:03:24.699044 9703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:03:24.699296 9703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:03:24.704207 9708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:24.704201 9709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:24.704208 9711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:24.704530 9703 server_base.cc:1034] running on GCE node
I20260924 03:03:24.704871 9703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:03:24.705139 9703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:03:24.706287 9703 hybrid_clock.cc:648] HybridClock initialized: now 1790219004706264 us; error 41 us; skew 500 ppm
I20260924 03:03:24.707683 9703 webserver.cc:460] Webserver started at http://127.0.98.1:32895/ using document root <none> and password file <none>
I20260924 03:03:24.707948 9703 fs_manager.cc:359] Metadata directory not provided
I20260924 03:03:24.708015 9703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:03:24.708160 9703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:03:24.709343 9703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/data/instance:
uuid: "82d9b71eb5f54d31b86dc8469b9218ab"
format_stamp: "Formatted at 2026-09-24 03:03:24 on dist-test-slave-wgsb"
server_key: "922ada3eeffea0a9f6d1c4e1e89fe743"
server_key_iv: "cc5f67e25cca1053c94420647c4d6a6a"
server_key_version: "encryptionkey@0"
I20260924 03:03:24.709702 9703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/wal/instance:
uuid: "82d9b71eb5f54d31b86dc8469b9218ab"
format_stamp: "Formatted at 2026-09-24 03:03:24 on dist-test-slave-wgsb"
server_key: "922ada3eeffea0a9f6d1c4e1e89fe743"
server_key_iv: "cc5f67e25cca1053c94420647c4d6a6a"
server_key_version: "encryptionkey@0"
I20260924 03:03:24.711040 9703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:03:24.712062 9717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:24.712302 9703 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 03:03:24.712410 9703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "82d9b71eb5f54d31b86dc8469b9218ab"
format_stamp: "Formatted at 2026-09-24 03:03:24 on dist-test-slave-wgsb"
server_key: "922ada3eeffea0a9f6d1c4e1e89fe743"
server_key_iv: "cc5f67e25cca1053c94420647c4d6a6a"
server_key_version: "encryptionkey@0"
I20260924 03:03:24.712491 9703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:03:24.740094 9703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:03:24.740434 9703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:03:24.740600 9703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:03:24.740844 9703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:03:24.741187 9703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:03:24.741242 9703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:24.741277 9703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:03:24.741304 9703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:24.747545 9703 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34301
I20260924 03:03:24.747654 9830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34301 every 8 connection(s)
I20260924 03:03:24.747997 9703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:03:24.748631 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 9703
I20260924 03:03:24.748697 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:03:24.748927 392 external_mini_cluster.cc:1428] Setting key b800f014c5d48a83dcfbeecbc2b5cd69
I20260924 03:03:24.749930 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43639
--builtin_ntp_servers=127.0.98.20:38893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 03:03:24.754093 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:24.749118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34153 (local address 127.0.98.62:43639)
0924 03:03:24.749162 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:24.749165 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:24.749178 (+ 13us) server_negotiation.cc:407] Connection header received
0924 03:03:24.749780 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:24.749781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:24.749794 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:24.749880 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:24.750620 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:24.751354 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:24.752435 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:24.752632 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:24.752718 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:24.753763 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:24.753764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:24.753765 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:03:24.753770 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:24.753792 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:03:24.753796 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:24.753853 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:24.753980 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 03:03:24.754008 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:03:24.754331 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:24.748831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43639 (local address 127.0.98.1:34153)
0924 03:03:24.749013 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:24.749063 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:24.749681 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:24.749930 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:24.749934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:24.749941 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:24.750508 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:24.750512 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:24.751458 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:24.751459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:24.752333 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:24.752335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:24.752736 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:24.752737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:24.752880 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:24.753552 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:03:24.753552 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:24.753591 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:03:24.753618 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:03:24.753868 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:03:24.753874 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:03:24.753877 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:03:24.753979 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 03:03:24.754102 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
I20260924 03:03:24.755306 9831 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43639
I20260924 03:03:24.755431 9831 heartbeater.cc:464] Registering TS with master...
I20260924 03:03:24.755688 9831 heartbeater.cc:511] Master 127.0.98.62:43639 requested a full tablet report, sending...
I20260924 03:03:24.756343 9643 ts_manager.cc:194] Registered new tserver with Master: 82d9b71eb5f54d31b86dc8469b9218ab (127.0.98.1:34301)
I20260924 03:03:24.757397 9643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:34153
W20260924 03:03:24.831915 9834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:03:24.832118 9834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:03:24.832137 9834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:03:24.833205 9834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:03:24.833238 9834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:03:24.833261 9834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:03:24.834510 9834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:03:24.834685 9834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:03:24.834908 9834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:03:24.840173 9840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:24.840194 9839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:24.840166 9842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:24.840387 9834 server_base.cc:1034] running on GCE node
I20260924 03:03:24.840675 9834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:03:24.840883 9834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:03:24.842039 9834 hybrid_clock.cc:648] HybridClock initialized: now 1790219004842024 us; error 28 us; skew 500 ppm
I20260924 03:03:24.843394 9834 webserver.cc:460] Webserver started at http://127.0.98.2:40447/ using document root <none> and password file <none>
I20260924 03:03:24.843684 9834 fs_manager.cc:359] Metadata directory not provided
I20260924 03:03:24.843736 9834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:03:24.843868 9834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:03:24.844921 9834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/data/instance:
uuid: "74efe3a87b40449fb5ac38067d677a7a"
format_stamp: "Formatted at 2026-09-24 03:03:24 on dist-test-slave-wgsb"
server_key: "dbdf74b6cdc39b9fb3003b8f9691b3bc"
server_key_iv: "07181b43d7b3f9532c20bb4038aeeb69"
server_key_version: "encryptionkey@0"
I20260924 03:03:24.845281 9834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/wal/instance:
uuid: "74efe3a87b40449fb5ac38067d677a7a"
format_stamp: "Formatted at 2026-09-24 03:03:24 on dist-test-slave-wgsb"
server_key: "dbdf74b6cdc39b9fb3003b8f9691b3bc"
server_key_iv: "07181b43d7b3f9532c20bb4038aeeb69"
server_key_version: "encryptionkey@0"
I20260924 03:03:24.846666 9834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:03:24.847712 9848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:24.847891 9834 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:03:24.847997 9834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "74efe3a87b40449fb5ac38067d677a7a"
format_stamp: "Formatted at 2026-09-24 03:03:24 on dist-test-slave-wgsb"
server_key: "dbdf74b6cdc39b9fb3003b8f9691b3bc"
server_key_iv: "07181b43d7b3f9532c20bb4038aeeb69"
server_key_version: "encryptionkey@0"
I20260924 03:03:24.848093 9834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:03:24.864651 9834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:03:24.864975 9834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:03:24.865129 9834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:03:24.865368 9834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:03:24.865710 9834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:03:24.865763 9834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:24.865799 9834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:03:24.865825 9834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:24.871994 9834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33893
I20260924 03:03:24.872035 9961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33893 every 8 connection(s)
I20260924 03:03:24.872380 9834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:03:24.874822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 9834
I20260924 03:03:24.874912 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:03:24.875101 392 external_mini_cluster.cc:1428] Setting key f1f55e9ce7e9b1b5992a11a5bcbb9996
I20260924 03:03:24.876156 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43639
--builtin_ntp_servers=127.0.98.20:38893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 03:03:24.878170 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:24.873146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45919 (local address 127.0.98.62:43639)
0924 03:03:24.873229 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:24.873232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:24.873590 (+ 358us) server_negotiation.cc:407] Connection header received
0924 03:03:24.874069 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:24.874069 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:24.874080 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:24.874133 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:24.874794 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:24.875496 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:24.876673 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:24.876993 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:24.877066 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:24.877936 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:24.877937 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:24.877937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:03:24.877941 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:24.877956 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:03:24.877960 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:24.877999 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:24.878090 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 03:03:24.878110 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 03:03:24.878453 9964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:24.873130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43639 (local address 127.0.98.2:45919)
0924 03:03:24.873399 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:24.873434 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:24.873984 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:24.874186 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:24.874190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:24.874198 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:24.874698 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:24.874702 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:24.875582 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:24.875583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:24.876589 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:24.876592 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:24.877076 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:24.877077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:24.877188 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:24.877775 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:03:24.877776 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:24.877816 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:03:24.877848 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:03:24.878015 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:03:24.878018 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:03:24.878021 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:03:24.878088 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 03:03:24.878203 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":136,"threads_started":1}
I20260924 03:03:24.878911 9962 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43639
I20260924 03:03:24.879024 9962 heartbeater.cc:464] Registering TS with master...
I20260924 03:03:24.879209 9962 heartbeater.cc:511] Master 127.0.98.62:43639 requested a full tablet report, sending...
I20260924 03:03:24.879665 9643 ts_manager.cc:194] Registered new tserver with Master: 74efe3a87b40449fb5ac38067d677a7a (127.0.98.2:33893)
I20260924 03:03:24.880244 9643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:45919
W20260924 03:03:24.962208 9965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:03:24.962430 9965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:03:24.962453 9965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:03:24.963954 9965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:03:24.964017 9965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:03:24.964046 9965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:03:24.965469 9965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:03:24.965757 9965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:03:24.966097 9965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:03:24.971360 9971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:24.971558 9965 server_base.cc:1034] running on GCE node
W20260924 03:03:24.971627 9970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:24.971360 9973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:24.972609 9965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:03:24.972826 9965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:03:24.974026 9965 hybrid_clock.cc:648] HybridClock initialized: now 1790219004974012 us; error 30 us; skew 500 ppm
I20260924 03:03:24.976353 9965 webserver.cc:460] Webserver started at http://127.0.98.3:40383/ using document root <none> and password file <none>
I20260924 03:03:24.976732 9965 fs_manager.cc:359] Metadata directory not provided
I20260924 03:03:24.976818 9965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:03:24.976984 9965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:03:24.978145 9965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/data/instance:
uuid: "090cb4a6c66b42a384693982130e030f"
format_stamp: "Formatted at 2026-09-24 03:03:24 on dist-test-slave-wgsb"
server_key: "dff4ac3dfc3eb9b8163822db3e062e14"
server_key_iv: "e3377c8e27cd36cfb3d94cc8bdc661e3"
server_key_version: "encryptionkey@0"
I20260924 03:03:24.978539 9965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/wal/instance:
uuid: "090cb4a6c66b42a384693982130e030f"
format_stamp: "Formatted at 2026-09-24 03:03:24 on dist-test-slave-wgsb"
server_key: "dff4ac3dfc3eb9b8163822db3e062e14"
server_key_iv: "e3377c8e27cd36cfb3d94cc8bdc661e3"
server_key_version: "encryptionkey@0"
I20260924 03:03:24.979998 9965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:03:24.980930 9979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:24.981258 9965 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:03:24.981400 9965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "090cb4a6c66b42a384693982130e030f"
format_stamp: "Formatted at 2026-09-24 03:03:24 on dist-test-slave-wgsb"
server_key: "dff4ac3dfc3eb9b8163822db3e062e14"
server_key_iv: "e3377c8e27cd36cfb3d94cc8bdc661e3"
server_key_version: "encryptionkey@0"
I20260924 03:03:24.981484 9965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:03:24.998560 9965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:03:24.998898 9965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:03:24.999053 9965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:03:24.999287 9965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:03:24.999699 9965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:03:24.999761 9965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:24.999804 9965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:03:24.999830 9965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:25.006012 9965 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37901
I20260924 03:03:25.006058 10092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37901 every 8 connection(s)
I20260924 03:03:25.006400 9965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:03:25.011831 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:25.007183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47507 (local address 127.0.98.62:43639)
0924 03:03:25.007242 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:25.007244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:25.007447 (+ 203us) server_negotiation.cc:407] Connection header received
0924 03:03:25.008081 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:25.008083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:25.008097 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:25.008162 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:25.008916 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:25.009418 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:25.010397 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:25.010628 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:25.010716 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:25.011516 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:25.011518 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:25.011518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:03:25.011523 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:25.011538 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:03:25.011543 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:25.011628 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:25.011723 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 03:03:25.011749 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 03:03:25.012100 10095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:25.007109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43639 (local address 127.0.98.3:47507)
0924 03:03:25.007317 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:25.007350 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:25.007950 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:25.008263 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:25.008266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:25.008275 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:25.008813 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:25.008817 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:25.009514 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:25.009515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:25.010303 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:25.010305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:25.010726 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:25.010726 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:25.010866 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:25.011378 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:03:25.011378 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:25.011405 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:03:25.011428 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:03:25.011632 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:03:25.011637 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:03:25.011640 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:03:25.011728 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 03:03:25.011828 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":112,"threads_started":1}
I20260924 03:03:25.012286 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 9965
I20260924 03:03:25.012364 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:03:25.012591 392 external_mini_cluster.cc:1428] Setting key f5de8617d61493923c1208f1142c043e
I20260924 03:03:25.012590 10093 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43639
I20260924 03:03:25.012688 10093 heartbeater.cc:464] Registering TS with master...
I20260924 03:03:25.012869 10093 heartbeater.cc:511] Master 127.0.98.62:43639 requested a full tablet report, sending...
I20260924 03:03:25.013237 9643 ts_manager.cc:194] Registered new tserver with Master: 090cb4a6c66b42a384693982130e030f (127.0.98.3:37901)
I20260924 03:03:25.013574 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:03:25.013741 9643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:47507
W20260924 03:03:25.015595 9682 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:25.015118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44070 (local address 127.0.98.62:43639)
0924 03:03:25.015155 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:25.015157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:25.015232 (+ 75us) server_negotiation.cc:407] Connection header received
0924 03:03:25.015289 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:25.015290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:25.015300 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:25.015376 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:25.015498 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44070: BlockingRecv error: recv got EOF from 127.0.0.1:44070 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:03:25.015622 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 9703
I20260924 03:03:25.020330 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 9834
I20260924 03:03:25.024819 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 9965
I20260924 03:03:25.029899 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 9612
2026-09-24T03:03:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (633 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T03:03:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:03:25Z Disabled control of system clock
I20260924 03:03:25.056386 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35709
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 03:03:25.138298 10104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:03:25.138553 10104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:03:25.138599 10104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:03:25.139782 10104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:03:25.139855 10104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:03:25.139878 10104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:03:25.139907 10104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:03:25.139938 10104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:03:25.141283 10104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35709
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:03:25.141566 10104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:03:25.141872 10104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:03:25.146926 10112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:25.146945 10110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:25.147150 10104 server_base.cc:1034] running on GCE node
W20260924 03:03:25.146944 10109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:25.147486 10104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:03:25.147799 10104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:03:25.148986 10104 hybrid_clock.cc:648] HybridClock initialized: now 1790219005148972 us; error 43 us; skew 500 ppm
I20260924 03:03:25.150501 10104 webserver.cc:460] Webserver started at http://127.0.98.62:39767/ using document root <none> and password file <none>
I20260924 03:03:25.150835 10104 fs_manager.cc:359] Metadata directory not provided
I20260924 03:03:25.150905 10104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:03:25.151067 10104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:03:25.152271 10104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "87d174468b52437d9b6ec13b68b71936"
format_stamp: "Formatted at 2026-09-24 03:03:25 on dist-test-slave-wgsb"
server_key: "05b00b8bb5e902a6854f37f0f1b17be5"
server_key_iv: "7cd0407001a182eac50c34a68a0529ab"
server_key_version: "encryptionkey@0"
I20260924 03:03:25.152637 10104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "87d174468b52437d9b6ec13b68b71936"
format_stamp: "Formatted at 2026-09-24 03:03:25 on dist-test-slave-wgsb"
server_key: "05b00b8bb5e902a6854f37f0f1b17be5"
server_key_iv: "7cd0407001a182eac50c34a68a0529ab"
server_key_version: "encryptionkey@0"
I20260924 03:03:25.154047 10104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:03:25.155005 10118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:25.155223 10104 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:03:25.155334 10104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "87d174468b52437d9b6ec13b68b71936"
format_stamp: "Formatted at 2026-09-24 03:03:25 on dist-test-slave-wgsb"
server_key: "05b00b8bb5e902a6854f37f0f1b17be5"
server_key_iv: "7cd0407001a182eac50c34a68a0529ab"
server_key_version: "encryptionkey@0"
I20260924 03:03:25.155408 10104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:03:25.158823 10104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:03:25.159104 10104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:03:25.159243 10104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:03:25.163182 10104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35709
I20260924 03:03:25.163306 10170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35709 every 8 connection(s)
I20260924 03:03:25.163568 10104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:03:25.164285 10171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:03:25.167855 10171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87d174468b52437d9b6ec13b68b71936: Bootstrap starting.
I20260924 03:03:25.168458 10171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87d174468b52437d9b6ec13b68b71936: Neither blocks nor log segments found. Creating new log.
I20260924 03:03:25.168797 10171 log.cc:824] T 00000000000000000000000000000000 P 87d174468b52437d9b6ec13b68b71936: Log is configured to *not* fsync() on all Append() calls
I20260924 03:03:25.169641 10171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87d174468b52437d9b6ec13b68b71936: No bootstrap required, opened a new log
I20260924 03:03:25.170627 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 10104
I20260924 03:03:25.170737 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:03:25.170915 392 external_mini_cluster.cc:1428] Setting key 2f9a21a19fc3288caf651ddadb9b51cf
I20260924 03:03:25.171507 10171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87d174468b52437d9b6ec13b68b71936 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d174468b52437d9b6ec13b68b71936" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35709 } }
I20260924 03:03:25.171739 10171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87d174468b52437d9b6ec13b68b71936 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:03:25.171787 10171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87d174468b52437d9b6ec13b68b71936 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87d174468b52437d9b6ec13b68b71936, State: Initialized, Role: FOLLOWER
I20260924 03:03:25.171909 10171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87d174468b52437d9b6ec13b68b71936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d174468b52437d9b6ec13b68b71936" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35709 } }
I20260924 03:03:25.171988 10171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87d174468b52437d9b6ec13b68b71936 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:03:25.172016 10171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87d174468b52437d9b6ec13b68b71936 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:03:25.172055 10171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87d174468b52437d9b6ec13b68b71936 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:03:25.172773 10171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87d174468b52437d9b6ec13b68b71936 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d174468b52437d9b6ec13b68b71936" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35709 } }
I20260924 03:03:25.172883 10171 leader_election.cc:302] T 00000000000000000000000000000000 P 87d174468b52437d9b6ec13b68b71936 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87d174468b52437d9b6ec13b68b71936; no voters:
I20260924 03:03:25.173035 10171 leader_election.cc:288] T 00000000000000000000000000000000 P 87d174468b52437d9b6ec13b68b71936 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:03:25.173089 10176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87d174468b52437d9b6ec13b68b71936 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:03:25.173233 10176 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87d174468b52437d9b6ec13b68b71936 [term 1 LEADER]: Becoming Leader. State: Replica: 87d174468b52437d9b6ec13b68b71936, State: Running, Role: LEADER
I20260924 03:03:25.173305 10171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87d174468b52437d9b6ec13b68b71936 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:03:25.173317 10176 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87d174468b52437d9b6ec13b68b71936 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d174468b52437d9b6ec13b68b71936" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35709 } }
I20260924 03:03:25.173707 10176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87d174468b52437d9b6ec13b68b71936 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87d174468b52437d9b6ec13b68b71936. Latest consensus state: current_term: 1 leader_uuid: "87d174468b52437d9b6ec13b68b71936" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d174468b52437d9b6ec13b68b71936" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35709 } } }
I20260924 03:03:25.173789 10176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87d174468b52437d9b6ec13b68b71936 [sys.catalog]: This master's current role is: LEADER
I20260924 03:03:25.173945 10177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87d174468b52437d9b6ec13b68b71936 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87d174468b52437d9b6ec13b68b71936" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d174468b52437d9b6ec13b68b71936" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35709 } } }
I20260924 03:03:25.174010 10183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:03:25.174012 10177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87d174468b52437d9b6ec13b68b71936 [sys.catalog]: This master's current role is: LEADER
I20260924 03:03:25.174355 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:25.171467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34692 (local address 127.0.98.62:35709)
0924 03:03:25.171758 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:25.171768 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:25.171803 (+ 35us) server_negotiation.cc:407] Connection header received
0924 03:03:25.171987 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:25.171994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:25.172042 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:25.172196 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:25.173548 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:25.173552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:25.173552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:03:25.173556 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:25.173599 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:03:25.173613 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:25.173669 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:25.173968 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 03:03:25.174153 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":193,"threads_started":1}
I20260924 03:03:25.174510 10183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:03:25.176672 10183 catalog_manager.cc:1347] Generated new cluster ID: 7695c89cf9644d0c8992899314e8072a
I20260924 03:03:25.176724 10183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:03:25.184577 10183 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:03:25.184741 10183 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:03:25.190127 10183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87d174468b52437d9b6ec13b68b71936: Generated new TSK 0
I20260924 03:03:25.190336 10183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:03:25.199524 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35709
--builtin_ntp_servers=127.0.98.20:40787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 03:03:25.283066 10195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:03:25.283294 10195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:03:25.283318 10195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:03:25.284613 10195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:03:25.284657 10195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:03:25.284684 10195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:03:25.286077 10195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:03:25.286356 10195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:03:25.286679 10195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:03:25.291956 10203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:25.292032 10200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:25.292116 10195 server_base.cc:1034] running on GCE node
W20260924 03:03:25.291926 10201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:25.292354 10195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:03:25.292616 10195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:03:25.293809 10195 hybrid_clock.cc:648] HybridClock initialized: now 1790219005293792 us; error 34 us; skew 500 ppm
I20260924 03:03:25.295368 10195 webserver.cc:460] Webserver started at http://127.0.98.1:37917/ using document root <none> and password file <none>
I20260924 03:03:25.295783 10195 fs_manager.cc:359] Metadata directory not provided
I20260924 03:03:25.295869 10195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:03:25.296028 10195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:03:25.297221 10195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/data/instance:
uuid: "0751caa91ddd4381a00af7bb4c0337ed"
format_stamp: "Formatted at 2026-09-24 03:03:25 on dist-test-slave-wgsb"
server_key: "b8579d525e2ccf69370230532f1d0bb4"
server_key_iv: "4b16fcc19e79a9f5afc4949b0c080c34"
server_key_version: "encryptionkey@0"
I20260924 03:03:25.297631 10195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0751caa91ddd4381a00af7bb4c0337ed"
format_stamp: "Formatted at 2026-09-24 03:03:25 on dist-test-slave-wgsb"
server_key: "b8579d525e2ccf69370230532f1d0bb4"
server_key_iv: "4b16fcc19e79a9f5afc4949b0c080c34"
server_key_version: "encryptionkey@0"
I20260924 03:03:25.299103 10195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:03:25.299979 10209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:25.300212 10195 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 03:03:25.300309 10195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "0751caa91ddd4381a00af7bb4c0337ed"
format_stamp: "Formatted at 2026-09-24 03:03:25 on dist-test-slave-wgsb"
server_key: "b8579d525e2ccf69370230532f1d0bb4"
server_key_iv: "4b16fcc19e79a9f5afc4949b0c080c34"
server_key_version: "encryptionkey@0"
I20260924 03:03:25.300377 10195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:03:25.305334 10195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:03:25.305634 10195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:03:25.305763 10195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:03:25.305965 10195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:03:25.306295 10195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:03:25.306336 10195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:25.306365 10195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:03:25.306386 10195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:25.312651 10195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36331
I20260924 03:03:25.312742 10322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36331 every 8 connection(s)
I20260924 03:03:25.312970 10195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:03:25.314495 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 10195
I20260924 03:03:25.314605 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:03:25.314814 392 external_mini_cluster.cc:1428] Setting key 927db7787406e5431d281a790537219e
I20260924 03:03:25.316224 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35709
--builtin_ntp_servers=127.0.98.20:40787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 03:03:25.316224 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:25.313648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51925 (local address 127.0.98.62:35709)
0924 03:03:25.313715 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:25.313717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:25.313927 (+ 210us) server_negotiation.cc:407] Connection header received
0924 03:03:25.314481 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:25.314482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:25.314530 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:25.314610 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:25.315936 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:25.315937 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:25.315938 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:03:25.315942 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:25.315958 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:03:25.315962 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:25.316035 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:25.316139 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 03:03:25.316155 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 03:03:25.316457 10325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:25.313624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35709 (local address 127.0.98.1:51925)
0924 03:03:25.313805 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:25.313839 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:25.314370 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:25.314706 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:25.314710 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:25.314718 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:25.315782 (+ 1064us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:03:25.315782 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:25.315834 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:03:25.315870 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:03:25.316034 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:03:25.316039 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:03:25.316042 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:03:25.316148 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 03:03:25.316209 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260924 03:03:25.316907 10323 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35709
I20260924 03:03:25.317005 10323 heartbeater.cc:464] Registering TS with master...
I20260924 03:03:25.317193 10323 heartbeater.cc:511] Master 127.0.98.62:35709 requested a full tablet report, sending...
I20260924 03:03:25.317754 10135 ts_manager.cc:194] Registered new tserver with Master: 0751caa91ddd4381a00af7bb4c0337ed (127.0.98.1:36331)
W20260924 03:03:25.399224 10326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:03:25.399482 10326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:03:25.399519 10326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:03:25.400733 10326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:03:25.400816 10326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:03:25.400863 10326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:03:25.402184 10326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:03:25.402374 10326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:03:25.402619 10326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:03:25.407770 10331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:25.407840 10334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:25.407963 10326 server_base.cc:1034] running on GCE node
W20260924 03:03:25.407797 10332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:25.408245 10326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:03:25.408449 10326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:03:25.409709 10326 hybrid_clock.cc:648] HybridClock initialized: now 1790219005409684 us; error 34 us; skew 500 ppm
I20260924 03:03:25.411087 10326 webserver.cc:460] Webserver started at http://127.0.98.2:39573/ using document root <none> and password file <none>
I20260924 03:03:25.411350 10326 fs_manager.cc:359] Metadata directory not provided
I20260924 03:03:25.411422 10326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:03:25.411682 10326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:03:25.412832 10326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/data/instance:
uuid: "cb2aa63cc0464e84a4a0a03987b7e067"
format_stamp: "Formatted at 2026-09-24 03:03:25 on dist-test-slave-wgsb"
server_key: "e32a010ffd7d54e83e9e65c770fafe8d"
server_key_iv: "4dcc589e686ca03f50f7c743681d6c33"
server_key_version: "encryptionkey@0"
I20260924 03:03:25.413192 10326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cb2aa63cc0464e84a4a0a03987b7e067"
format_stamp: "Formatted at 2026-09-24 03:03:25 on dist-test-slave-wgsb"
server_key: "e32a010ffd7d54e83e9e65c770fafe8d"
server_key_iv: "4dcc589e686ca03f50f7c743681d6c33"
server_key_version: "encryptionkey@0"
I20260924 03:03:25.414599 10326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:03:25.415423 10340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:25.415699 10326 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:03:25.415800 10326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "cb2aa63cc0464e84a4a0a03987b7e067"
format_stamp: "Formatted at 2026-09-24 03:03:25 on dist-test-slave-wgsb"
server_key: "e32a010ffd7d54e83e9e65c770fafe8d"
server_key_iv: "4dcc589e686ca03f50f7c743681d6c33"
server_key_version: "encryptionkey@0"
I20260924 03:03:25.415881 10326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:03:25.419216 10326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:03:25.419477 10326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:03:25.419656 10326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:03:25.419878 10326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:03:25.420185 10326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:03:25.420239 10326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:25.420279 10326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:03:25.420306 10326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:25.427809 10326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39381
I20260924 03:03:25.427879 10453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39381 every 8 connection(s)
I20260924 03:03:25.428275 10326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:03:25.431363 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 10326
I20260924 03:03:25.431497 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:03:25.431741 392 external_mini_cluster.cc:1428] Setting key c9002b25d7577ec214b44fed5ad0d4a7
I20260924 03:03:25.431867 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:25.429115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44979 (local address 127.0.98.62:35709)
0924 03:03:25.429185 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:25.429187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:25.429472 (+ 285us) server_negotiation.cc:407] Connection header received
0924 03:03:25.430045 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:25.430046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:25.430052 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:25.430104 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:25.431459 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:25.431460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:25.431460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:03:25.431465 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:25.431483 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:03:25.431488 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:25.431546 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:25.431759 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 03:03:25.431772 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 03:03:25.432112 10456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:25.429098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35709 (local address 127.0.98.2:44979)
0924 03:03:25.429316 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:25.429376 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:25.429932 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:25.430177 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:25.430181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:25.430191 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:25.431295 (+ 1104us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:03:25.431295 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:25.431324 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:03:25.431353 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:03:25.431648 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:03:25.431656 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:03:25.431660 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:03:25.431769 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 03:03:25.431843 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":115,"threads_started":1}
I20260924 03:03:25.432543 10454 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35709
I20260924 03:03:25.432682 10454 heartbeater.cc:464] Registering TS with master...
I20260924 03:03:25.432801 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35709
--builtin_ntp_servers=127.0.98.20:40787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 03:03:25.432853 10454 heartbeater.cc:511] Master 127.0.98.62:35709 requested a full tablet report, sending...
I20260924 03:03:25.433183 10135 ts_manager.cc:194] Registered new tserver with Master: cb2aa63cc0464e84a4a0a03987b7e067 (127.0.98.2:39381)
W20260924 03:03:25.515473 10457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:03:25.515790 10457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:03:25.515831 10457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:03:25.517076 10457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:03:25.517143 10457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:03:25.517186 10457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:03:25.518661 10457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:03:25.518920 10457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:03:25.519222 10457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:03:25.524398 10457 server_base.cc:1034] running on GCE node
W20260924 03:03:25.524372 10463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:25.524366 10462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:25.524379 10465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:25.524827 10457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:03:25.525113 10457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:03:25.526284 10457 hybrid_clock.cc:648] HybridClock initialized: now 1790219005526273 us; error 52 us; skew 500 ppm
I20260924 03:03:25.527670 10457 webserver.cc:460] Webserver started at http://127.0.98.3:37139/ using document root <none> and password file <none>
I20260924 03:03:25.527925 10457 fs_manager.cc:359] Metadata directory not provided
I20260924 03:03:25.527997 10457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:03:25.528156 10457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:03:25.529217 10457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/data/instance:
uuid: "2db9e2d39e354030a6226c9f2cc5eea2"
format_stamp: "Formatted at 2026-09-24 03:03:25 on dist-test-slave-wgsb"
server_key: "ca49b99b3df410432d33650ad56a41f7"
server_key_iv: "de72e163d6d35e5a2f20f7f4aeff5f03"
server_key_version: "encryptionkey@0"
I20260924 03:03:25.529565 10457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2db9e2d39e354030a6226c9f2cc5eea2"
format_stamp: "Formatted at 2026-09-24 03:03:25 on dist-test-slave-wgsb"
server_key: "ca49b99b3df410432d33650ad56a41f7"
server_key_iv: "de72e163d6d35e5a2f20f7f4aeff5f03"
server_key_version: "encryptionkey@0"
I20260924 03:03:25.530866 10457 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:03:25.531579 10471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:25.531809 10457 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:03:25.531904 10457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "2db9e2d39e354030a6226c9f2cc5eea2"
format_stamp: "Formatted at 2026-09-24 03:03:25 on dist-test-slave-wgsb"
server_key: "ca49b99b3df410432d33650ad56a41f7"
server_key_iv: "de72e163d6d35e5a2f20f7f4aeff5f03"
server_key_version: "encryptionkey@0"
I20260924 03:03:25.531991 10457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:03:25.535259 10457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:03:25.535504 10457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:03:25.535673 10457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:03:25.535882 10457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:03:25.536175 10457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:03:25.536227 10457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:25.536266 10457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:03:25.536296 10457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:25.542047 10457 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42379
I20260924 03:03:25.542099 10584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42379 every 8 connection(s)
I20260924 03:03:25.542389 10457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:03:25.545338 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:25.543098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35967 (local address 127.0.98.62:35709)
0924 03:03:25.543176 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:25.543178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:25.543309 (+ 131us) server_negotiation.cc:407] Connection header received
0924 03:03:25.543965 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:25.543966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:25.543971 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:25.544026 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:25.545114 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:25.545115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:25.545116 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:03:25.545120 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:25.545133 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:03:25.545137 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:25.545194 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:25.545270 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 03:03:25.545283 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 03:03:25.545487 10587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:25.542997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35709 (local address 127.0.98.3:35967)
0924 03:03:25.543183 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:25.543216 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:25.543866 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:25.544083 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:25.544086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:25.544094 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:25.544934 (+ 840us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:03:25.544934 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:25.544961 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:03:25.544986 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:03:25.545193 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:03:25.545195 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:03:25.545197 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:03:25.545247 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 03:03:25.545292 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
I20260924 03:03:25.546030 10585 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35709
I20260924 03:03:25.546113 10585 heartbeater.cc:464] Registering TS with master...
I20260924 03:03:25.546226 10585 heartbeater.cc:511] Master 127.0.98.62:35709 requested a full tablet report, sending...
I20260924 03:03:25.546516 10135 ts_manager.cc:194] Registered new tserver with Master: 2db9e2d39e354030a6226c9f2cc5eea2 (127.0.98.3:42379)
I20260924 03:03:25.548338 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 10457
I20260924 03:03:25.548421 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218948286999-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:03:25.548621 392 external_mini_cluster.cc:1428] Setting key e06393b117de3a6907194f20ff406bdd
I20260924 03:03:25.549479 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 03:03:25.551410 10175 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:03:25.550857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34702 (local address 127.0.98.62:35709)
0924 03:03:25.550911 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:25.550913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:25.551051 (+ 138us) server_negotiation.cc:407] Connection header received
0924 03:03:25.551139 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:25.551141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:25.551146 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:25.551209 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:25.551345 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34702: BlockingRecv error: recv got EOF from 127.0.0.1:34702 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 03:03:25.551479 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 10195
I20260924 03:03:25.556813 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 10326
I20260924 03:03:25.561291 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 10457
I20260924 03:03:25.565804 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 10104
2026-09-24T03:03:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (536 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:03:25 dist-test-slave-wgsb krb5kdc[10595](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:03:25 dist-test-slave-wgsb krb5kdc[10595](info): set up 2 sockets
Sep 24 03:03:25 dist-test-slave-wgsb krb5kdc[10595](info): commencing operation
krb5kdc: starting...
W20260924 03:03:26.628896 392 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:03:26 dist-test-slave-wgsb krb5kdc[10595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:03:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:03:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:03:26.750283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33681
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:03:26.832916 10608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:03:26.833132 10608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:03:26.833158 10608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:03:26.834277 10608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:03:26.834321 10608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:03:26.834338 10608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:03:26.834352 10608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:03:26.834367 10608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:03:26.835891 10608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:03:26.836086 10608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:03:26.836256 10608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:03:26.841132 10614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:26.841125 10616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:26.841121 10613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:26.841341 10608 server_base.cc:1034] running on GCE node
I20260924 03:03:26.841570 10608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:03:26.841886 10608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:03:26.843029 10608 hybrid_clock.cc:648] HybridClock initialized: now 1790219006843012 us; error 39 us; skew 500 ppm
Sep 24 03:03:26 dist-test-slave-wgsb krb5kdc[10595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:03:26.845427 10608 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:03:26.845883 10608 webserver.cc:460] Webserver started at http://127.0.98.62:33321/ using document root <none> and password file <none>
I20260924 03:03:26.846104 10608 fs_manager.cc:359] Metadata directory not provided
I20260924 03:03:26.846170 10608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:03:26.846331 10608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:03:26.847385 10608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "9b24cf64feab48429bee4ca220959de6"
format_stamp: "Formatted at 2026-09-24 03:03:26 on dist-test-slave-wgsb"
server_key: "c5c12b7ec9575a7be1a9521a3d6ad355"
server_key_iv: "76b4bb3075551ce98536386c26f43d03"
server_key_version: "encryptionkey@0"
I20260924 03:03:26.847783 10608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "9b24cf64feab48429bee4ca220959de6"
format_stamp: "Formatted at 2026-09-24 03:03:26 on dist-test-slave-wgsb"
server_key: "c5c12b7ec9575a7be1a9521a3d6ad355"
server_key_iv: "76b4bb3075551ce98536386c26f43d03"
server_key_version: "encryptionkey@0"
I20260924 03:03:26.849077 10608 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:03:26.849870 10623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:26.850049 10608 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:03:26.850147 10608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "9b24cf64feab48429bee4ca220959de6"
format_stamp: "Formatted at 2026-09-24 03:03:26 on dist-test-slave-wgsb"
server_key: "c5c12b7ec9575a7be1a9521a3d6ad355"
server_key_iv: "76b4bb3075551ce98536386c26f43d03"
server_key_version: "encryptionkey@0"
I20260924 03:03:26.850234 10608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:03:26.876538 10608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:03:26.878767 10608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:03:26.878939 10608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:03:26.883057 10608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33681
I20260924 03:03:26.883100 10675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33681 every 8 connection(s)
I20260924 03:03:26.883479 10608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:03:26.884591 10676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:03:26.885433 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 10608
I20260924 03:03:26.885543 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:03:26.885746 392 external_mini_cluster.cc:1428] Setting key efeb0154e37d7051cb8378301740f97f
I20260924 03:03:26.888123 10676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b24cf64feab48429bee4ca220959de6: Bootstrap starting.
I20260924 03:03:26.888901 10676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b24cf64feab48429bee4ca220959de6: Neither blocks nor log segments found. Creating new log.
I20260924 03:03:26.889242 10676 log.cc:824] T 00000000000000000000000000000000 P 9b24cf64feab48429bee4ca220959de6: Log is configured to *not* fsync() on all Append() calls
I20260924 03:03:26.890007 10676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b24cf64feab48429bee4ca220959de6: No bootstrap required, opened a new log
I20260924 03:03:26.892554 10676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b24cf64feab48429bee4ca220959de6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b24cf64feab48429bee4ca220959de6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33681 } }
I20260924 03:03:26.892700 10676 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b24cf64feab48429bee4ca220959de6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:03:26.892725 10676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b24cf64feab48429bee4ca220959de6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b24cf64feab48429bee4ca220959de6, State: Initialized, Role: FOLLOWER
I20260924 03:03:26.892876 10676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b24cf64feab48429bee4ca220959de6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b24cf64feab48429bee4ca220959de6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33681 } }
I20260924 03:03:26.893005 10676 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b24cf64feab48429bee4ca220959de6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:03:26.893059 10676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b24cf64feab48429bee4ca220959de6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:03:26.893146 10676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b24cf64feab48429bee4ca220959de6 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 03:03:26 dist-test-slave-wgsb krb5kdc[10595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:03:26.893816 10676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b24cf64feab48429bee4ca220959de6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b24cf64feab48429bee4ca220959de6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33681 } }
I20260924 03:03:26.893944 10676 leader_election.cc:302] T 00000000000000000000000000000000 P 9b24cf64feab48429bee4ca220959de6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b24cf64feab48429bee4ca220959de6; no voters:
I20260924 03:03:26.894163 10676 leader_election.cc:288] T 00000000000000000000000000000000 P 9b24cf64feab48429bee4ca220959de6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:03:26.894199 10681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b24cf64feab48429bee4ca220959de6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:03:26.894393 10681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b24cf64feab48429bee4ca220959de6 [term 1 LEADER]: Becoming Leader. State: Replica: 9b24cf64feab48429bee4ca220959de6, State: Running, Role: LEADER
I20260924 03:03:26.894454 10676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b24cf64feab48429bee4ca220959de6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:03:26.894484 10681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b24cf64feab48429bee4ca220959de6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b24cf64feab48429bee4ca220959de6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33681 } }
I20260924 03:03:26.894779 10681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b24cf64feab48429bee4ca220959de6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b24cf64feab48429bee4ca220959de6. Latest consensus state: current_term: 1 leader_uuid: "9b24cf64feab48429bee4ca220959de6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b24cf64feab48429bee4ca220959de6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33681 } } }
I20260924 03:03:26.894871 10681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b24cf64feab48429bee4ca220959de6 [sys.catalog]: This master's current role is: LEADER
I20260924 03:03:26.894789 10683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b24cf64feab48429bee4ca220959de6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b24cf64feab48429bee4ca220959de6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b24cf64feab48429bee4ca220959de6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33681 } } }
I20260924 03:03:26.895011 10683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b24cf64feab48429bee4ca220959de6 [sys.catalog]: This master's current role is: LEADER
I20260924 03:03:26.896029 10686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:03:26.896534 10686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:03:26.898087 10686 catalog_manager.cc:1347] Generated new cluster ID: 0416994652bd4ed78dc75bde84bb526d
I20260924 03:03:26.898136 10686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:03:26.898715 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:26.886345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47436 (local address 127.0.98.62:33681)
0924 03:03:26.886548 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:26.886558 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:26.886596 (+ 38us) server_negotiation.cc:407] Connection header received
0924 03:03:26.887212 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:26.887222 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:26.887292 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:26.887454 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:26.888066 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:26.889610 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:26.890422 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:26.891104 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:26.891229 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:26.894030 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:26.894035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:26.894035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:03:26.894048 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:26.896532 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:26.896997 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:26.897000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:26.897001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:26.897108 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:26.897334 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:26.897334 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:26.897335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:26.897660 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:26.898275 (+ 615us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:26.898353 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 03:03:26.898498 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":83,"threads_started":1}
I20260924 03:03:26.905993 10686 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:03:26.906617 10686 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:03:26.918005 10686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b24cf64feab48429bee4ca220959de6: Generated new TSK 0
I20260924 03:03:26.918231 10686 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:03:26.946581 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33681
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:03:27.029309 10702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:03:27.029518 10702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:03:27.029542 10702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:03:27.030779 10702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:03:27.030819 10702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:03:27.030848 10702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:03:27.032299 10702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:03:27.032529 10702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:03:27.032789 10702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:03:27.037758 10708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:27.037793 10707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:27.038017 10710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:27.038060 10702 server_base.cc:1034] running on GCE node
I20260924 03:03:27.038280 10702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:03:27.038537 10702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:03:27.039670 10702 hybrid_clock.cc:648] HybridClock initialized: now 1790219007039658 us; error 37 us; skew 500 ppm
Sep 24 03:03:27 dist-test-slave-wgsb krb5kdc[10595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:03:27.042501 10702 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:03:27.042937 10702 webserver.cc:460] Webserver started at http://127.0.98.1:42045/ using document root <none> and password file <none>
I20260924 03:03:27.043159 10702 fs_manager.cc:359] Metadata directory not provided
I20260924 03:03:27.043232 10702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:03:27.043380 10702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:03:27.044934 10702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-0/data/instance:
uuid: "a1e3d08f9f464d609247e9b5faa4582d"
format_stamp: "Formatted at 2026-09-24 03:03:27 on dist-test-slave-wgsb"
server_key: "c826053034b6b4e62a49bf6bcedc39d3"
server_key_iv: "d557762640bee2b0fac91c7a9c39efd1"
server_key_version: "encryptionkey@0"
I20260924 03:03:27.045301 10702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a1e3d08f9f464d609247e9b5faa4582d"
format_stamp: "Formatted at 2026-09-24 03:03:27 on dist-test-slave-wgsb"
server_key: "c826053034b6b4e62a49bf6bcedc39d3"
server_key_iv: "d557762640bee2b0fac91c7a9c39efd1"
server_key_version: "encryptionkey@0"
I20260924 03:03:27.046663 10702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:03:27.047429 10717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:27.047763 10702 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:03:27.047866 10702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "a1e3d08f9f464d609247e9b5faa4582d"
format_stamp: "Formatted at 2026-09-24 03:03:27 on dist-test-slave-wgsb"
server_key: "c826053034b6b4e62a49bf6bcedc39d3"
server_key_iv: "d557762640bee2b0fac91c7a9c39efd1"
server_key_version: "encryptionkey@0"
I20260924 03:03:27.047955 10702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:03:27.090026 10702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:03:27.092489 10702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:03:27.092664 10702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:03:27.092926 10702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:03:27.093245 10702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:03:27.093299 10702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:27.093339 10702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:03:27.093381 10702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:27.099536 10702 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44523
I20260924 03:03:27.099742 10830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44523 every 8 connection(s)
I20260924 03:03:27.099962 10702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:03:27.101805 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 10702
I20260924 03:03:27.101909 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:03:27.102089 392 external_mini_cluster.cc:1428] Setting key e20c2f1a1e9c9ecc00639541e4f613f9
Sep 24 03:03:27 dist-test-slave-wgsb krb5kdc[10595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 03:03:27.111145 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:27.100741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42925 (local address 127.0.98.62:33681)
0924 03:03:27.100814 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:27.100816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:27.101018 (+ 202us) server_negotiation.cc:407] Connection header received
0924 03:03:27.101583 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:27.101585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:27.101599 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:27.101668 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:27.102983 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.103794 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.104933 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.105165 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.105266 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:27.107916 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:27.107918 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:27.107918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:03:27.107926 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:27.109490 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:27.110030 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:27.110031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:27.110032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:27.110096 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:27.110411 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:27.110412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:27.110412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:27.110566 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:27.110647 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:27.110930 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 03:03:27.111033 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 03:03:27.111343 10833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:27.100700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33681 (local address 127.0.98.1:42925)
0924 03:03:27.100888 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:27.100929 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:27.101458 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:27.101726 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:27.101730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:27.102563 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:27.102883 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:27.102889 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.103912 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.103913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:27.104820 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:27.104823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.105268 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.105269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:27.105413 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:27.106016 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:27.106025 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:27.107816 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:03:27.109690 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:03:27.109697 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:03:27.109699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:03:27.109936 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:03:27.110172 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:03:27.110173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:03:27.110173 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:03:27.110321 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:03:27.110649 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:03:27.110656 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:03:27.110800 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 03:03:27.110910 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 03:03:27.111096 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":110,"threads_started":1}
I20260924 03:03:27.111860 10831 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33681
I20260924 03:03:27.111975 10831 heartbeater.cc:464] Registering TS with master...
I20260924 03:03:27.112218 10831 heartbeater.cc:511] Master 127.0.98.62:33681 requested a full tablet report, sending...
I20260924 03:03:27.112872 10640 ts_manager.cc:194] Registered new tserver with Master: a1e3d08f9f464d609247e9b5faa4582d (127.0.98.1:44523)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 03:03:27.113617 10640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:03:27.130396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33681
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:03:27.214447 10836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:03:27.214659 10836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:03:27.214685 10836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:03:27.215909 10836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:03:27.215960 10836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:03:27.215988 10836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:03:27.217439 10836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:03:27.217732 10836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:03:27.217995 10836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:03:27.223273 10842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:27.223273 10841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:27.223373 10836 server_base.cc:1034] running on GCE node
W20260924 03:03:27.223354 10844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:27.223873 10836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:03:27.224124 10836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:03:27.225287 10836 hybrid_clock.cc:648] HybridClock initialized: now 1790219007225285 us; error 39 us; skew 500 ppm
Sep 24 03:03:27 dist-test-slave-wgsb krb5kdc[10595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:03:27.227877 10836 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:03:27.228322 10836 webserver.cc:460] Webserver started at http://127.0.98.2:44753/ using document root <none> and password file <none>
I20260924 03:03:27.228547 10836 fs_manager.cc:359] Metadata directory not provided
I20260924 03:03:27.228615 10836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:03:27.228766 10836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:03:27.229817 10836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-1/data/instance:
uuid: "75f91c92202c4610979cb02d3716f4a1"
format_stamp: "Formatted at 2026-09-24 03:03:27 on dist-test-slave-wgsb"
server_key: "2d466aff256718c5f76eb90ce1843d64"
server_key_iv: "de0bd47d81cc86c57768b3256f3a7a23"
server_key_version: "encryptionkey@0"
I20260924 03:03:27.230181 10836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-1/wal/instance:
uuid: "75f91c92202c4610979cb02d3716f4a1"
format_stamp: "Formatted at 2026-09-24 03:03:27 on dist-test-slave-wgsb"
server_key: "2d466aff256718c5f76eb90ce1843d64"
server_key_iv: "de0bd47d81cc86c57768b3256f3a7a23"
server_key_version: "encryptionkey@0"
I20260924 03:03:27.231539 10836 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:03:27.232430 10851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:27.232648 10836 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:03:27.232786 10836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "75f91c92202c4610979cb02d3716f4a1"
format_stamp: "Formatted at 2026-09-24 03:03:27 on dist-test-slave-wgsb"
server_key: "2d466aff256718c5f76eb90ce1843d64"
server_key_iv: "de0bd47d81cc86c57768b3256f3a7a23"
server_key_version: "encryptionkey@0"
I20260924 03:03:27.232867 10836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:03:27.249536 10836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:03:27.251875 10836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:03:27.252058 10836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:03:27.252316 10836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:03:27.252658 10836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:03:27.252712 10836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:27.252763 10836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:03:27.252794 10836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:27.259083 10836 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34629
I20260924 03:03:27.259132 10964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34629 every 8 connection(s)
I20260924 03:03:27.259476 10836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:03:27.265596 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 10836
I20260924 03:03:27.265722 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-1/wal/instance
Sep 24 03:03:27 dist-test-slave-wgsb krb5kdc[10595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:03:27.265918 392 external_mini_cluster.cc:1428] Setting key 076c40d50f4d32efdd449326cbae174e
I20260924 03:03:27.269586 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:27.260328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43373 (local address 127.0.98.62:33681)
0924 03:03:27.260397 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:27.260399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:27.260496 (+ 97us) server_negotiation.cc:407] Connection header received
0924 03:03:27.261010 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:27.261011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:27.261022 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:27.261086 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:27.262048 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.262545 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.263503 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.263706 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.263811 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:27.266452 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:27.266453 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:27.266453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:03:27.266460 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:27.268020 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:27.268609 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:27.268610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:27.268611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:27.268669 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:27.268937 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:27.268938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:27.268938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:27.269057 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:27.269127 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:27.269416 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 03:03:27.269495 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 03:03:27.269917 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:27.260192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33681 (local address 127.0.98.2:43373)
0924 03:03:27.260357 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:27.260402 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:27.260920 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:27.261160 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:27.261163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:27.261762 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:27.261957 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:27.261961 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.262627 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.262628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:27.263422 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:27.263423 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.263827 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.263828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:27.263985 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:27.264536 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:27.264546 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:27.266366 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:03:27.268126 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:03:27.268134 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:03:27.268136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:03:27.268534 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:03:27.268747 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:03:27.268748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:03:27.268749 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:03:27.268865 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:03:27.269131 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:03:27.269137 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:03:27.269282 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 03:03:27.269386 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 03:03:27.269547 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260924 03:03:27.270437 10965 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33681
I20260924 03:03:27.270558 10965 heartbeater.cc:464] Registering TS with master...
I20260924 03:03:27.270767 10965 heartbeater.cc:511] Master 127.0.98.62:33681 requested a full tablet report, sending...
I20260924 03:03:27.271176 10640 ts_manager.cc:194] Registered new tserver with Master: 75f91c92202c4610979cb02d3716f4a1 (127.0.98.2:34629)
I20260924 03:03:27.271576 10640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43373
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:03:27.294641 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33681
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:03:27.384404 10970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:03:27.384644 10970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:03:27.384671 10970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:03:27.385887 10970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:03:27.385938 10970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:03:27.385967 10970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:03:27.387530 10970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:03:27.387809 10970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:03:27.388036 10970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:03:27.393477 10975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:27.393498 10970 server_base.cc:1034] running on GCE node
W20260924 03:03:27.393467 10978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:03:27.393468 10976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:03:27.393913 10970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:03:27.394209 10970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:03:27.395344 10970 hybrid_clock.cc:648] HybridClock initialized: now 1790219007395369 us; error 67 us; skew 500 ppm
Sep 24 03:03:27 dist-test-slave-wgsb krb5kdc[10595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:03:27.398175 10970 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:03:27.398631 10970 webserver.cc:460] Webserver started at http://127.0.98.3:45933/ using document root <none> and password file <none>
I20260924 03:03:27.398867 10970 fs_manager.cc:359] Metadata directory not provided
I20260924 03:03:27.398941 10970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:03:27.399080 10970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:03:27.400251 10970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-2/data/instance:
uuid: "ec1bd34d8a254feb93e461c73cb747cb"
format_stamp: "Formatted at 2026-09-24 03:03:27 on dist-test-slave-wgsb"
server_key: "3a614f313e01f82b6640d6a444276a0f"
server_key_iv: "c989f603670a6f58b9822752754c89d0"
server_key_version: "encryptionkey@0"
I20260924 03:03:27.400672 10970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ec1bd34d8a254feb93e461c73cb747cb"
format_stamp: "Formatted at 2026-09-24 03:03:27 on dist-test-slave-wgsb"
server_key: "3a614f313e01f82b6640d6a444276a0f"
server_key_iv: "c989f603670a6f58b9822752754c89d0"
server_key_version: "encryptionkey@0"
I20260924 03:03:27.402041 10970 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:03:27.402832 10985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:27.403019 10970 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:03:27.403115 10970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "ec1bd34d8a254feb93e461c73cb747cb"
format_stamp: "Formatted at 2026-09-24 03:03:27 on dist-test-slave-wgsb"
server_key: "3a614f313e01f82b6640d6a444276a0f"
server_key_iv: "c989f603670a6f58b9822752754c89d0"
server_key_version: "encryptionkey@0"
I20260924 03:03:27.403196 10970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:03:27.419584 10970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:03:27.421576 10970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:03:27.421733 10970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:03:27.421983 10970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:03:27.422293 10970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:03:27.422345 10970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:27.422389 10970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:03:27.422417 10970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:03:27.428174 10970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40881
I20260924 03:03:27.428236 11098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40881 every 8 connection(s)
I20260924 03:03:27.428573 10970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:03:27.430218 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 10970
I20260924 03:03:27.430325 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:03:27.430514 392 external_mini_cluster.cc:1428] Setting key 104b651b142bd2014c6afc8e6e0d4025
Sep 24 03:03:27 dist-test-slave-wgsb krb5kdc[10595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:03:27.438597 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:27.429331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39917 (local address 127.0.98.62:33681)
0924 03:03:27.429393 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:27.429396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:27.429578 (+ 182us) server_negotiation.cc:407] Connection header received
0924 03:03:27.430133 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:27.430134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:27.430145 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:27.430202 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:27.431388 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.431974 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.432954 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.433142 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.433213 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:27.435645 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:27.435647 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:27.435647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:03:27.435657 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:27.437225 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:27.437667 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:27.437669 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:27.437669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:27.437726 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:27.437964 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:27.437965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:27.437965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:27.438082 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:27.438156 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:27.438421 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 03:03:27.438500 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 03:03:27.438824 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:27.429317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33681 (local address 127.0.98.3:39917)
0924 03:03:27.429464 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:27.429502 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:27.430033 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:27.430269 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:27.430274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:27.431043 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:03:27.431309 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:27.431315 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.432062 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.432063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:27.432858 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:27.432860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.433217 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.433218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:27.433345 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:27.433842 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:03:27.433850 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:03:27.435498 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:03:27.437347 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:03:27.437354 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:03:27.437356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:03:27.437580 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:03:27.437793 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:03:27.437794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:03:27.437794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:03:27.437898 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:03:27.438147 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:03:27.438153 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:03:27.438295 (+ 142us) client_negotiation.cc:769] Sending connection context
0924 03:03:27.438403 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 03:03:27.438561 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260924 03:03:27.439317 11099 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33681
I20260924 03:03:27.439397 11099 heartbeater.cc:464] Registering TS with master...
I20260924 03:03:27.439643 11099 heartbeater.cc:511] Master 127.0.98.62:33681 requested a full tablet report, sending...
I20260924 03:03:27.440068 10640 ts_manager.cc:194] Registered new tserver with Master: ec1bd34d8a254feb93e461c73cb747cb (127.0.98.3:40881)
I20260924 03:03:27.440409 10640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39917
I20260924 03:03:27.442086 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:03:27.448530 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:27.443263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47440 (local address 127.0.98.62:33681)
0924 03:03:27.443325 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:27.443327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:27.443385 (+ 58us) server_negotiation.cc:407] Connection header received
0924 03:03:27.443442 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:27.443460 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:27.443468 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:27.443522 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:03:27.444096 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.444556 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.445213 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.445386 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.445437 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:27.446493 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:03:27.446494 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:03:27.446494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:03:27.446500 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:03:27.447631 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:27.447926 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:27.447927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:27.447927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:27.447974 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:03:27.448131 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:03:27.448131 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:03:27.448131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:03:27.448268 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:03:27.448339 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:27.448420 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 03:03:27.448471 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 03:03:27.450251 10640 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:03:27.452294 10640 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:03:27.459136 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:27.454901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34629 (local address 127.0.0.1:50314)
0924 03:03:27.455172 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:27.455183 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:27.455238 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:27.455688 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:27.455689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:27.455691 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:03:27.455748 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:27.455749 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.457143 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.457144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:27.458128 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:27.458131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.458873 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.458874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:27.458943 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:27.458947 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:03:27.458998 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 03:03:27.459038 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":88,"threads_started":1}
I20260924 03:03:27.459147 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:27.455019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50314 (local address 127.0.98.2:34629)
0924 03:03:27.455494 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:27.455498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:27.455507 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:03:27.455544 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:27.455546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:27.455562 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:27.455879 (+ 317us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:03:27.456019 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.457039 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.458250 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.458780 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.458859 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:27.458914 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:27.459021 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 03:03:27.459060 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":95,"threads_started":1}
I20260924 03:03:27.459374 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:27.454753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40881 (local address 127.0.0.1:40640)
0924 03:03:27.455063 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:27.455104 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:27.455180 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:27.455464 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:27.455468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:27.455471 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:03:27.455541 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:27.455543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.456942 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.456943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:27.458000 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:27.458002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.458758 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.458759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:27.458851 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:27.458867 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 03:03:27.458936 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 03:03:27.458992 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":69,"threads_started":1}
I20260924 03:03:27.459663 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:27.454872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40640 (local address 127.0.98.3:40881)
0924 03:03:27.455122 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:27.455127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:27.455292 (+ 165us) server_negotiation.cc:407] Connection header received
0924 03:03:27.455337 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:27.455339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:27.455357 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:27.455438 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:03:27.455705 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.456841 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.458086 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.458631 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.458690 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:27.458732 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:27.459083 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 03:03:27.459127 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":86,"threads_started":1}
I20260924 03:03:27.459973 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:27.454962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44523 (local address 127.0.0.1:50534)
0924 03:03:27.455314 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:27.455323 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:27.455354 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:27.455547 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:27.455548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:27.455550 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:03:27.455692 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:27.455696 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.456685 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.456687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:27.457989 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:27.457992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.459709 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.459711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:27.459791 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:27.459795 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:03:27.459843 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 03:03:27.459884 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":61,"threads_started":1}
I20260924 03:03:27.459906 10899 tablet_service.cc:1467] Processing CreateTablet for tablet aa4aee80ad5b4ace88110248c8e2cc4a (DEFAULT_TABLE table=test-table [id=18f0b21d844146d3b4c3dc556595890d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:03:27.460281 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:27.455158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50534 (local address 127.0.98.1:44523)
0924 03:03:27.455346 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:27.455352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:27.455361 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:03:27.455419 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:27.455421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:27.455436 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:27.455539 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:03:27.455760 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.456570 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.458762 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.459543 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.459626 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:27.459677 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:27.460150 (+ 473us) server_negotiation.cc:299] Negotiation successful
0924 03:03:27.460193 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":94,"threads_started":1}
I20260924 03:03:27.460305 10899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa4aee80ad5b4ace88110248c8e2cc4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:03:27.460283 11033 tablet_service.cc:1467] Processing CreateTablet for tablet aa4aee80ad5b4ace88110248c8e2cc4a (DEFAULT_TABLE table=test-table [id=18f0b21d844146d3b4c3dc556595890d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:03:27.460618 11033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa4aee80ad5b4ace88110248c8e2cc4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:03:27.460875 10765 tablet_service.cc:1467] Processing CreateTablet for tablet aa4aee80ad5b4ace88110248c8e2cc4a (DEFAULT_TABLE table=test-table [id=18f0b21d844146d3b4c3dc556595890d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:03:27.461172 10765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa4aee80ad5b4ace88110248c8e2cc4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:03:27.463447 11116 tablet_bootstrap.cc:492] T aa4aee80ad5b4ace88110248c8e2cc4a P ec1bd34d8a254feb93e461c73cb747cb: Bootstrap starting.
I20260924 03:03:27.463972 11117 tablet_bootstrap.cc:492] T aa4aee80ad5b4ace88110248c8e2cc4a P 75f91c92202c4610979cb02d3716f4a1: Bootstrap starting.
I20260924 03:03:27.464167 11116 tablet_bootstrap.cc:654] T aa4aee80ad5b4ace88110248c8e2cc4a P ec1bd34d8a254feb93e461c73cb747cb: Neither blocks nor log segments found. Creating new log.
I20260924 03:03:27.464453 11118 tablet_bootstrap.cc:492] T aa4aee80ad5b4ace88110248c8e2cc4a P a1e3d08f9f464d609247e9b5faa4582d: Bootstrap starting.
I20260924 03:03:27.464634 11117 tablet_bootstrap.cc:654] T aa4aee80ad5b4ace88110248c8e2cc4a P 75f91c92202c4610979cb02d3716f4a1: Neither blocks nor log segments found. Creating new log.
I20260924 03:03:27.464782 11116 log.cc:824] T aa4aee80ad5b4ace88110248c8e2cc4a P ec1bd34d8a254feb93e461c73cb747cb: Log is configured to *not* fsync() on all Append() calls
I20260924 03:03:27.465214 11118 tablet_bootstrap.cc:654] T aa4aee80ad5b4ace88110248c8e2cc4a P a1e3d08f9f464d609247e9b5faa4582d: Neither blocks nor log segments found. Creating new log.
I20260924 03:03:27.465188 11117 log.cc:824] T aa4aee80ad5b4ace88110248c8e2cc4a P 75f91c92202c4610979cb02d3716f4a1: Log is configured to *not* fsync() on all Append() calls
I20260924 03:03:27.465688 11116 tablet_bootstrap.cc:492] T aa4aee80ad5b4ace88110248c8e2cc4a P ec1bd34d8a254feb93e461c73cb747cb: No bootstrap required, opened a new log
I20260924 03:03:27.465821 11116 ts_tablet_manager.cc:1397] T aa4aee80ad5b4ace88110248c8e2cc4a P ec1bd34d8a254feb93e461c73cb747cb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:03:27.465807 11118 log.cc:824] T aa4aee80ad5b4ace88110248c8e2cc4a P a1e3d08f9f464d609247e9b5faa4582d: Log is configured to *not* fsync() on all Append() calls
I20260924 03:03:27.466189 11117 tablet_bootstrap.cc:492] T aa4aee80ad5b4ace88110248c8e2cc4a P 75f91c92202c4610979cb02d3716f4a1: No bootstrap required, opened a new log
I20260924 03:03:27.466271 11117 ts_tablet_manager.cc:1397] T aa4aee80ad5b4ace88110248c8e2cc4a P 75f91c92202c4610979cb02d3716f4a1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:03:27.466711 11118 tablet_bootstrap.cc:492] T aa4aee80ad5b4ace88110248c8e2cc4a P a1e3d08f9f464d609247e9b5faa4582d: No bootstrap required, opened a new log
I20260924 03:03:27.466787 11118 ts_tablet_manager.cc:1397] T aa4aee80ad5b4ace88110248c8e2cc4a P a1e3d08f9f464d609247e9b5faa4582d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:03:27.467819 11116 raft_consensus.cc:348] T aa4aee80ad5b4ace88110248c8e2cc4a P ec1bd34d8a254feb93e461c73cb747cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec1bd34d8a254feb93e461c73cb747cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40881 } } peers { permanent_uuid: "a1e3d08f9f464d609247e9b5faa4582d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44523 } } peers { permanent_uuid: "75f91c92202c4610979cb02d3716f4a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34629 } }
I20260924 03:03:27.467984 11116 raft_consensus.cc:374] T aa4aee80ad5b4ace88110248c8e2cc4a P ec1bd34d8a254feb93e461c73cb747cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:03:27.468029 11116 raft_consensus.cc:729] T aa4aee80ad5b4ace88110248c8e2cc4a P ec1bd34d8a254feb93e461c73cb747cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec1bd34d8a254feb93e461c73cb747cb, State: Initialized, Role: FOLLOWER
I20260924 03:03:27.468175 11116 consensus_queue.cc:260] T aa4aee80ad5b4ace88110248c8e2cc4a P ec1bd34d8a254feb93e461c73cb747cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec1bd34d8a254feb93e461c73cb747cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40881 } } peers { permanent_uuid: "a1e3d08f9f464d609247e9b5faa4582d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44523 } } peers { permanent_uuid: "75f91c92202c4610979cb02d3716f4a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34629 } }
I20260924 03:03:27.468286 11117 raft_consensus.cc:348] T aa4aee80ad5b4ace88110248c8e2cc4a P 75f91c92202c4610979cb02d3716f4a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec1bd34d8a254feb93e461c73cb747cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40881 } } peers { permanent_uuid: "a1e3d08f9f464d609247e9b5faa4582d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44523 } } peers { permanent_uuid: "75f91c92202c4610979cb02d3716f4a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34629 } }
I20260924 03:03:27.468456 11117 raft_consensus.cc:374] T aa4aee80ad5b4ace88110248c8e2cc4a P 75f91c92202c4610979cb02d3716f4a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:03:27.468477 11116 ts_tablet_manager.cc:1428] T aa4aee80ad5b4ace88110248c8e2cc4a P ec1bd34d8a254feb93e461c73cb747cb: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260924 03:03:27.468501 11117 raft_consensus.cc:729] T aa4aee80ad5b4ace88110248c8e2cc4a P 75f91c92202c4610979cb02d3716f4a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f91c92202c4610979cb02d3716f4a1, State: Initialized, Role: FOLLOWER
I20260924 03:03:27.468653 11099 heartbeater.cc:503] Master 127.0.98.62:33681 was elected leader, sending a full tablet report...
I20260924 03:03:27.468640 11117 consensus_queue.cc:260] T aa4aee80ad5b4ace88110248c8e2cc4a P 75f91c92202c4610979cb02d3716f4a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec1bd34d8a254feb93e461c73cb747cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40881 } } peers { permanent_uuid: "a1e3d08f9f464d609247e9b5faa4582d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44523 } } peers { permanent_uuid: "75f91c92202c4610979cb02d3716f4a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34629 } }
I20260924 03:03:27.468890 11117 ts_tablet_manager.cc:1428] T aa4aee80ad5b4ace88110248c8e2cc4a P 75f91c92202c4610979cb02d3716f4a1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:03:27.469015 10965 heartbeater.cc:503] Master 127.0.98.62:33681 was elected leader, sending a full tablet report...
I20260924 03:03:27.468775 11118 raft_consensus.cc:348] T aa4aee80ad5b4ace88110248c8e2cc4a P a1e3d08f9f464d609247e9b5faa4582d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec1bd34d8a254feb93e461c73cb747cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40881 } } peers { permanent_uuid: "a1e3d08f9f464d609247e9b5faa4582d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44523 } } peers { permanent_uuid: "75f91c92202c4610979cb02d3716f4a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34629 } }
I20260924 03:03:27.469134 11118 raft_consensus.cc:374] T aa4aee80ad5b4ace88110248c8e2cc4a P a1e3d08f9f464d609247e9b5faa4582d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:03:27.469185 11118 raft_consensus.cc:729] T aa4aee80ad5b4ace88110248c8e2cc4a P a1e3d08f9f464d609247e9b5faa4582d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1e3d08f9f464d609247e9b5faa4582d, State: Initialized, Role: FOLLOWER
I20260924 03:03:27.469406 11118 consensus_queue.cc:260] T aa4aee80ad5b4ace88110248c8e2cc4a P a1e3d08f9f464d609247e9b5faa4582d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec1bd34d8a254feb93e461c73cb747cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40881 } } peers { permanent_uuid: "a1e3d08f9f464d609247e9b5faa4582d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44523 } } peers { permanent_uuid: "75f91c92202c4610979cb02d3716f4a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34629 } }
I20260924 03:03:27.469781 11118 ts_tablet_manager.cc:1428] T aa4aee80ad5b4ace88110248c8e2cc4a P a1e3d08f9f464d609247e9b5faa4582d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:03:27.469800 10831 heartbeater.cc:503] Master 127.0.98.62:33681 was elected leader, sending a full tablet report...
W20260924 03:03:27.510181 10966 tablet.cc:2369] T aa4aee80ad5b4ace88110248c8e2cc4a P 75f91c92202c4610979cb02d3716f4a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:03:27.600791 10832 tablet.cc:2369] T aa4aee80ad5b4ace88110248c8e2cc4a P a1e3d08f9f464d609247e9b5faa4582d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:03:27.633970 11124 raft_consensus.cc:482] T aa4aee80ad5b4ace88110248c8e2cc4a P a1e3d08f9f464d609247e9b5faa4582d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:03:27.634133 11124 raft_consensus.cc:504] T aa4aee80ad5b4ace88110248c8e2cc4a P a1e3d08f9f464d609247e9b5faa4582d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec1bd34d8a254feb93e461c73cb747cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40881 } } peers { permanent_uuid: "a1e3d08f9f464d609247e9b5faa4582d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44523 } } peers { permanent_uuid: "75f91c92202c4610979cb02d3716f4a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34629 } }
I20260924 03:03:27.634652 11124 leader_election.cc:288] T aa4aee80ad5b4ace88110248c8e2cc4a P a1e3d08f9f464d609247e9b5faa4582d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec1bd34d8a254feb93e461c73cb747cb (127.0.98.3:40881), 75f91c92202c4610979cb02d3716f4a1 (127.0.98.2:34629)
I20260924 03:03:27.637995 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:27.634933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54055 (local address 127.0.98.2:34629)
0924 03:03:27.634982 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:27.634985 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:27.635162 (+ 177us) server_negotiation.cc:407] Connection header received
0924 03:03:27.635197 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:27.635198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:27.635211 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:27.635261 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:03:27.635478 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.636161 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.637224 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.637628 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.637714 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:27.637724 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:27.637877 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 03:03:27.637924 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 03:03:27.638177 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:27.634936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55739 (local address 127.0.98.3:40881)
0924 03:03:27.634982 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:27.634985 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:27.635078 (+ 93us) server_negotiation.cc:407] Connection header received
0924 03:03:27.635259 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:27.635260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:27.635274 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:27.635327 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:03:27.635573 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.636150 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.637236 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.637654 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.637745 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:27.637755 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:27.638035 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 03:03:27.638095 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 03:03:27.638021 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:27.634861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34629 (local address 127.0.98.1:54055)
0924 03:03:27.634990 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:27.635003 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:27.635076 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:27.635285 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:27.635286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:27.635288 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:03:27.635399 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:27.635403 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.636243 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.636244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:27.637139 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:27.637141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.637714 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.637715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:27.637791 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:27.637795 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:03:27.637868 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 03:03:27.637909 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260924 03:03:27.638314 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:27.634847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40881 (local address 127.0.98.1:55739)
0924 03:03:27.634990 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:27.635003 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:03:27.635172 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:03:27.635373 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:03:27.635374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:03:27.635376 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:03:27.635441 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:27.635443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.636252 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.636253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:27.637140 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:03:27.637142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.637748 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.637749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:03:27.637823 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:27.637826 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:03:27.637901 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 03:03:27.637953 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":34,"thread_start_us":68,"threads_started":1}
I20260924 03:03:27.638652 10919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa4aee80ad5b4ace88110248c8e2cc4a" candidate_uuid: "a1e3d08f9f464d609247e9b5faa4582d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f91c92202c4610979cb02d3716f4a1" is_pre_election: true
I20260924 03:03:27.638633 11053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa4aee80ad5b4ace88110248c8e2cc4a" candidate_uuid: "a1e3d08f9f464d609247e9b5faa4582d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec1bd34d8a254feb93e461c73cb747cb" is_pre_election: true
I20260924 03:03:27.638799 11053 raft_consensus.cc:2454] T aa4aee80ad5b4ace88110248c8e2cc4a P ec1bd34d8a254feb93e461c73cb747cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1e3d08f9f464d609247e9b5faa4582d in term 0.
I20260924 03:03:27.638799 10919 raft_consensus.cc:2454] T aa4aee80ad5b4ace88110248c8e2cc4a P 75f91c92202c4610979cb02d3716f4a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1e3d08f9f464d609247e9b5faa4582d in term 0.
I20260924 03:03:27.639060 10718 leader_election.cc:302] T aa4aee80ad5b4ace88110248c8e2cc4a P a1e3d08f9f464d609247e9b5faa4582d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1e3d08f9f464d609247e9b5faa4582d, ec1bd34d8a254feb93e461c73cb747cb; no voters:
I20260924 03:03:27.639247 11124 raft_consensus.cc:2781] T aa4aee80ad5b4ace88110248c8e2cc4a P a1e3d08f9f464d609247e9b5faa4582d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:03:27.639294 11124 raft_consensus.cc:482] T aa4aee80ad5b4ace88110248c8e2cc4a P a1e3d08f9f464d609247e9b5faa4582d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:03:27.639313 11124 raft_consensus.cc:3037] T aa4aee80ad5b4ace88110248c8e2cc4a P a1e3d08f9f464d609247e9b5faa4582d [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:03:27.640080 11124 raft_consensus.cc:504] T aa4aee80ad5b4ace88110248c8e2cc4a P a1e3d08f9f464d609247e9b5faa4582d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec1bd34d8a254feb93e461c73cb747cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40881 } } peers { permanent_uuid: "a1e3d08f9f464d609247e9b5faa4582d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44523 } } peers { permanent_uuid: "75f91c92202c4610979cb02d3716f4a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34629 } }
I20260924 03:03:27.640228 11124 leader_election.cc:288] T aa4aee80ad5b4ace88110248c8e2cc4a P a1e3d08f9f464d609247e9b5faa4582d [CANDIDATE]: Term 1 election: Requested vote from peers ec1bd34d8a254feb93e461c73cb747cb (127.0.98.3:40881), 75f91c92202c4610979cb02d3716f4a1 (127.0.98.2:34629)
I20260924 03:03:27.640358 11053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa4aee80ad5b4ace88110248c8e2cc4a" candidate_uuid: "a1e3d08f9f464d609247e9b5faa4582d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec1bd34d8a254feb93e461c73cb747cb"
I20260924 03:03:27.640398 10919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa4aee80ad5b4ace88110248c8e2cc4a" candidate_uuid: "a1e3d08f9f464d609247e9b5faa4582d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f91c92202c4610979cb02d3716f4a1"
I20260924 03:03:27.640441 11053 raft_consensus.cc:3037] T aa4aee80ad5b4ace88110248c8e2cc4a P ec1bd34d8a254feb93e461c73cb747cb [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:03:27.640462 10919 raft_consensus.cc:3037] T aa4aee80ad5b4ace88110248c8e2cc4a P 75f91c92202c4610979cb02d3716f4a1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:03:27.641343 11053 raft_consensus.cc:2454] T aa4aee80ad5b4ace88110248c8e2cc4a P ec1bd34d8a254feb93e461c73cb747cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1e3d08f9f464d609247e9b5faa4582d in term 1.
I20260924 03:03:27.641454 10919 raft_consensus.cc:2454] T aa4aee80ad5b4ace88110248c8e2cc4a P 75f91c92202c4610979cb02d3716f4a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1e3d08f9f464d609247e9b5faa4582d in term 1.
I20260924 03:03:27.641530 10718 leader_election.cc:302] T aa4aee80ad5b4ace88110248c8e2cc4a P a1e3d08f9f464d609247e9b5faa4582d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1e3d08f9f464d609247e9b5faa4582d, ec1bd34d8a254feb93e461c73cb747cb; no voters:
I20260924 03:03:27.641639 11124 raft_consensus.cc:2781] T aa4aee80ad5b4ace88110248c8e2cc4a P a1e3d08f9f464d609247e9b5faa4582d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:03:27.641808 11124 raft_consensus.cc:686] T aa4aee80ad5b4ace88110248c8e2cc4a P a1e3d08f9f464d609247e9b5faa4582d [term 1 LEADER]: Becoming Leader. State: Replica: a1e3d08f9f464d609247e9b5faa4582d, State: Running, Role: LEADER
I20260924 03:03:27.641919 11124 consensus_queue.cc:237] T aa4aee80ad5b4ace88110248c8e2cc4a P a1e3d08f9f464d609247e9b5faa4582d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec1bd34d8a254feb93e461c73cb747cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40881 } } peers { permanent_uuid: "a1e3d08f9f464d609247e9b5faa4582d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44523 } } peers { permanent_uuid: "75f91c92202c4610979cb02d3716f4a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34629 } }
I20260924 03:03:27.642930 10639 catalog_manager.cc:5495] T aa4aee80ad5b4ace88110248c8e2cc4a P a1e3d08f9f464d609247e9b5faa4582d reported cstate change: term changed from 0 to 1, leader changed from <none> to a1e3d08f9f464d609247e9b5faa4582d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a1e3d08f9f464d609247e9b5faa4582d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec1bd34d8a254feb93e461c73cb747cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1e3d08f9f464d609247e9b5faa4582d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75f91c92202c4610979cb02d3716f4a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34629 } health_report { overall_health: UNKNOWN } } }
W20260924 03:03:27.679355 11100 tablet.cc:2369] T aa4aee80ad5b4ace88110248c8e2cc4a P ec1bd34d8a254feb93e461c73cb747cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:03:27.685333 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:27.682776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50542 (local address 127.0.98.1:44523)
0924 03:03:27.682836 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 03:03:27.682838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:03:27.682847 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:03:27.682958 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:03:27.682959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:03:27.682979 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:03:27.683058 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:03:27.683232 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.683725 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.684506 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:03:27.684705 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:03:27.684792 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:03:27.684987 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:03:27.685089 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:03:27.685186 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 03:03:27.685209 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 03:03:27.685230 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 03:03:27.687266 10919 raft_consensus.cc:1261] T aa4aee80ad5b4ace88110248c8e2cc4a P 75f91c92202c4610979cb02d3716f4a1 [term 1 FOLLOWER]: Refusing update from remote peer a1e3d08f9f464d609247e9b5faa4582d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:03:27.687280 11053 raft_consensus.cc:1261] T aa4aee80ad5b4ace88110248c8e2cc4a P ec1bd34d8a254feb93e461c73cb747cb [term 1 FOLLOWER]: Refusing update from remote peer a1e3d08f9f464d609247e9b5faa4582d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:03:27.687685 11127 consensus_queue.cc:1035] T aa4aee80ad5b4ace88110248c8e2cc4a P a1e3d08f9f464d609247e9b5faa4582d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f91c92202c4610979cb02d3716f4a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:03:27.687847 11124 consensus_queue.cc:1035] T aa4aee80ad5b4ace88110248c8e2cc4a P a1e3d08f9f464d609247e9b5faa4582d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec1bd34d8a254feb93e461c73cb747cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:03:27.692055 11132 mvcc.cc:205] Tried to move back new op lower bound from 7332737055484096512 to 7332737055301984256. Current Snapshot: MvccSnapshot[applied={T|T < 7332737055484096512}]
I20260924 03:03:27.692577 11131 mvcc.cc:205] Tried to move back new op lower bound from 7332737055484096512 to 7332737055301984256. Current Snapshot: MvccSnapshot[applied={T|T < 7332737055484096512 or (T in {7332737055484096512})}]
I20260924 03:03:27.694010 10639 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:03:27.694118 10639 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:03:27.694958 10639 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b24cf64feab48429bee4ca220959de6: Sending DeleteTablet for 3 replicas of tablet aa4aee80ad5b4ace88110248c8e2cc4a
I20260924 03:03:27.695423 10899 tablet_service.cc:1514] Processing DeleteTablet for tablet aa4aee80ad5b4ace88110248c8e2cc4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:03:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50314
I20260924 03:03:27.695434 11033 tablet_service.cc:1514] Processing DeleteTablet for tablet aa4aee80ad5b4ace88110248c8e2cc4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:03:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40640
I20260924 03:03:27.695699 11139 tablet_replica.cc:330] T aa4aee80ad5b4ace88110248c8e2cc4a P ec1bd34d8a254feb93e461c73cb747cb: stopping tablet replica
I20260924 03:03:27.695762 11140 tablet_replica.cc:330] T aa4aee80ad5b4ace88110248c8e2cc4a P 75f91c92202c4610979cb02d3716f4a1: stopping tablet replica
I20260924 03:03:27.695806 11139 raft_consensus.cc:2229] T aa4aee80ad5b4ace88110248c8e2cc4a P ec1bd34d8a254feb93e461c73cb747cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:03:27.695889 11140 raft_consensus.cc:2229] T aa4aee80ad5b4ace88110248c8e2cc4a P 75f91c92202c4610979cb02d3716f4a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:03:27.695863 10765 tablet_service.cc:1514] Processing DeleteTablet for tablet aa4aee80ad5b4ace88110248c8e2cc4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:03:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50534
I20260924 03:03:27.695914 11139 raft_consensus.cc:2258] T aa4aee80ad5b4ace88110248c8e2cc4a P ec1bd34d8a254feb93e461c73cb747cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:03:27.695984 11140 raft_consensus.cc:2258] T aa4aee80ad5b4ace88110248c8e2cc4a P 75f91c92202c4610979cb02d3716f4a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:03:27.696107 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 10702
I20260924 03:03:27.697005 11140 ts_tablet_manager.cc:1905] T aa4aee80ad5b4ace88110248c8e2cc4a P 75f91c92202c4610979cb02d3716f4a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:03:27.697005 11139 ts_tablet_manager.cc:1905] T aa4aee80ad5b4ace88110248c8e2cc4a P ec1bd34d8a254feb93e461c73cb747cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:03:27.699422 11139 ts_tablet_manager.cc:1918] T aa4aee80ad5b4ace88110248c8e2cc4a P ec1bd34d8a254feb93e461c73cb747cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:03:27.699424 11140 ts_tablet_manager.cc:1918] T aa4aee80ad5b4ace88110248c8e2cc4a P 75f91c92202c4610979cb02d3716f4a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:03:27.699503 11139 log.cc:1196] T aa4aee80ad5b4ace88110248c8e2cc4a P ec1bd34d8a254feb93e461c73cb747cb: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-2/wal/wals/aa4aee80ad5b4ace88110248c8e2cc4a
I20260924 03:03:27.699506 11140 log.cc:1196] T aa4aee80ad5b4ace88110248c8e2cc4a P 75f91c92202c4610979cb02d3716f4a1: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218948286999-392-0/minicluster-data/ts-1/wal/wals/aa4aee80ad5b4ace88110248c8e2cc4a
I20260924 03:03:27.699862 11139 ts_tablet_manager.cc:1939] T aa4aee80ad5b4ace88110248c8e2cc4a P ec1bd34d8a254feb93e461c73cb747cb: Deleting consensus metadata
I20260924 03:03:27.699884 11140 ts_tablet_manager.cc:1939] T aa4aee80ad5b4ace88110248c8e2cc4a P 75f91c92202c4610979cb02d3716f4a1: Deleting consensus metadata
I20260924 03:03:27.700307 10624 catalog_manager.cc:4841] TS ec1bd34d8a254feb93e461c73cb747cb (127.0.98.3:40881): tablet aa4aee80ad5b4ace88110248c8e2cc4a (table test-table [id=18f0b21d844146d3b4c3dc556595890d]) successfully deleted
I20260924 03:03:27.700440 10625 catalog_manager.cc:4841] TS 75f91c92202c4610979cb02d3716f4a1 (127.0.98.2:34629): tablet aa4aee80ad5b4ace88110248c8e2cc4a (table test-table [id=18f0b21d844146d3b4c3dc556595890d]) successfully deleted
W20260924 03:03:27.703138 10624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44523 (error 108)
I20260924 03:03:27.703491 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 10836
I20260924 03:03:27.703661 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:03:27.703517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44523 (local address 127.0.0.1:50556)
0924 03:03:27.703571 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 03:03:27.703622 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 03:03:27.704159 10624 catalog_manager.cc:4584] TS a1e3d08f9f464d609247e9b5faa4582d (127.0.98.1:44523): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aa4aee80ad5b4ace88110248c8e2cc4a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44523: connect: Connection refused (error 111)
I20260924 03:03:27.709553 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 10970
I20260924 03:03:27.714216 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 10608
2026-09-24T03:03:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2148 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:03:27 dist-test-slave-wgsb krb5kdc[11145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:03:27 dist-test-slave-wgsb krb5kdc[11145](info): set up 2 sockets
Sep 24 03:03:27 dist-test-slave-wgsb krb5kdc[11145](info): commencing operation
krb5kdc: starting...
W20260924 03:03:28.785204 392 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:03:28 dist-test-slave-wgsb krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:03:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:03:28Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 03:03:28.970769 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskptPCbv/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/postgres -l logfile start
2026-09-24 03:03:31.761 UTC [11175] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 03:03:31.761 UTC [11175] LOG: listening on IPv4 address "127.0.98.20", port 56155
2026-09-24 03:03:31.765 UTC [11175] LOG: listening on Unix socket "/tmp/.s.PGSQL.56155"
2026-09-24 03:03:31.788 UTC [11178] LOG: database system was shut down at 2026-09-24 03:03:30 UTC
2026-09-24 03:03:31.792 UTC [11175] LOG: database system is ready to accept connections
I20260924 03:03:32.740536 392 mini_postgres.cc:96] Postgres bound to 56155
2026-09-24 03:03:32.746 UTC [11186] FATAL: database "slave" does not exist
127.0.98.20:56155 - accepting connections
I20260924 03:03:32.747110 392 mini_ranger.cc:161] Starting Ranger...
I20260924 03:03:32.761691 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 03:03:33.278632 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 03:03:33.278738 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ranger-admin
2026-09-24 03:03:33,549 [I] DB FLAVOR :POSTGRES
2026-09-24 03:03:33,550 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 03:03:33,550 [I] Checking connection..
2026-09-24 03:03:33,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 03:03:33,809 [I] Checking connection passed.
2026-09-24 03:03:33,809 [I] --------- Verifying version history table ---------
2026-09-24 03:03:33,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:03:34,068 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 03:03:34,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:03:34,329 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 03:03:34,329 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 03:03:34,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56155/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 03:03:34.580 UTC [11250] WARNING: there is no transaction in progress
2026-09-24 03:03:34,586 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 03:03:34,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:03:34,867 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 03:03:34,867 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 03:03:34,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 03:03:35,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 03:03:35,365 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wgsb.c.gcp-upstream.internal','N') ;"
2026-09-24 03:03:35,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 03:03:35,846 [I] Table x_portal_user does not exist in database ranger
2026-09-24 03:03:35,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 03:03:36,081 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 03:03:36,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 03:03:36,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 03:03:36,631 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 03:03:36,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56155/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 03:03:37.670 UTC [11358] WARNING: there is no transaction in progress
2026-09-24 03:03:37.687 UTC [11358] WARNING: there is no transaction in progress
2026-09-24 03:03:37.705 UTC [11358] WARNING: there is no transaction in progress
2026-09-24 03:03:37.722 UTC [11358] WARNING: there is no transaction in progress
2026-09-24 03:03:37.739 UTC [11358] WARNING: there is no transaction in progress
2026-09-24 03:03:37.758 UTC [11358] WARNING: there is no transaction in progress
2026-09-24 03:03:37.824 UTC [11358] WARNING: there is no transaction in progress
2026-09-24 03:03:37.832 UTC [11358] WARNING: there is no transaction in progress
2026-09-24 03:03:37.848 UTC [11358] WARNING: there is no transaction in progress
2026-09-24 03:03:37.859 UTC [11358] WARNING: there is no transaction in progress
2026-09-24 03:03:37.870 UTC [11358] WARNING: there is no transaction in progress
2026-09-24 03:03:37.885 UTC [11358] WARNING: there is no transaction in progress
2026-09-24 03:03:37.896 UTC [11358] WARNING: there is no transaction in progress
2026-09-24 03:03:37.903 UTC [11358] WARNING: there is no transaction in progress
2026-09-24 03:03:37.910 UTC [11358] WARNING: there is no transaction in progress
2026-09-24 03:03:37.918 UTC [11358] WARNING: there is no transaction in progress
2026-09-24 03:03:38,624 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 03:03:38,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 03:03:38,882 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 03:03:38,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 03:03:39,151 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 03:03:39,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 03:03:39,405 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 03:03:39,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 03:03:39,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 03:03:39,890 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wgsb.c.gcp-upstream.internal';"
2026-09-24 03:03:40,116 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 03:03:40,116 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 03:03:40,366 [I] DB_PATCHES have already been applied
I20260924 03:03:40.372350 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskptPCbv/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 03:03:40.372414 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 03:03:40.375454 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:52057
Sep 24, 2026 3:03:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:03:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:03:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 3:03:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 3:03:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 3:03:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 3:03:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 03:03:41.481446 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.734s user 0.000s sys 0.009s
Sep 24, 2026 3:04:01 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 3:04:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 3:04:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 3:04:01 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 3:04:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 3:04:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 3:04:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 3:04:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 3:04:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 3:04:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 3:04:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 3:04:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 3:04:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 3:04:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 3:04:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 3:04:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 3:04:02 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 3:04:02 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 03:04:03.471829 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 03:04:03 dist-test-slave-wgsb krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219043, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 03:04:03.560979 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 03:04:03.571014 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 03:04:04.631019 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 03:04:04.631106 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ranger-kms
I20260924 03:04:04.631131 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:56155
2026-09-24 03:04:04,807 [I] DB FLAVOR :POSTGRES
2026-09-24 03:04:04,807 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 03:04:04,808 [I] Checking connection
2026-09-24 03:04:04,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56155/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 03:04:05,124 [I] connection success
2026-09-24 03:04:05,125 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 03:04:05,125 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 03:04:05,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56155/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 03:04:05,409 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 03:04:05,409 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 03:04:05,410 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 03:04:05,410 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56155/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 03:04:05,698 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 03:04:06.163712 392 mini_ranger_kms.cc:310] Created kms service
I20260924 03:04:06.375039 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 03:04:06.418653 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 03:04:06 dist-test-slave-wgsb krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219043, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:04:06.840772 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 03:04:06.840890 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 03:04:06.840914 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 03:04:06.844812 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11538
I20260924 03:04:06.844959 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:39849
Sep 24, 2026 3:04:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:04:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:04:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 3:04:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 3:04:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 3:04:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 03:04:09.131845 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.571s user 0.000s sys 0.158s
I20260924 03:04:09.131963 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 03:04:09.132048 392 mini_ranger_kms.cc:401] 127.0.98.20:39849/kms/v1/keys
Sep 24 03:04:09 dist-test-slave-wgsb krb5kdc[11145](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790219049, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 03:04:09 dist-test-slave-wgsb krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219049, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 03:04:10 dist-test-slave-wgsb krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219049, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 03:04:12 dist-test-slave-wgsb krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:04:12.471000 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33581
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33581
--ranger_config_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39849/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:04:12.557672 11589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:04:12.557945 11589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:04:12.557979 11589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:04:12.559165 11589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:04:12.559253 11589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:04:12.559281 11589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:04:12.559311 11589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:04:12.559331 11589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:04:12.561049 11589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:39849/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33581
--ranger_config_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:04:12.561343 11589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:04:12.561582 11589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:04:12.567272 11597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:04:12.567281 11594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:04:12.567471 11589 server_base.cc:1034] running on GCE node
W20260924 03:04:12.567521 11595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:04:12.567766 11589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:04:12.568106 11589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:04:12.569300 11589 hybrid_clock.cc:648] HybridClock initialized: now 1790219052569271 us; error 51 us; skew 500 ppm
Sep 24 03:04:12 dist-test-slave-wgsb krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:04:12.572597 11589 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:04:12.573151 11589 webserver.cc:460] Webserver started at http://127.0.98.62:45903/ using document root <none> and password file <none>
I20260924 03:04:12.573473 11589 fs_manager.cc:359] Metadata directory not provided
I20260924 03:04:12.573537 11589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:04:12.573670 11589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:04:12 dist-test-slave-wgsb krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:04:12.639930 11589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "0d225ea56a8240d088be362c1402d7e6"
format_stamp: "Formatted at 2026-09-24 03:04:12 on dist-test-slave-wgsb"
server_key: "096863aaffb5a52a55836f0dc6f5df17"
server_key_iv: "cce97ea32eba3fff29504f4b33e8a09b"
server_key_version: "kuduclusterkey@0"
I20260924 03:04:12.640386 11589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "0d225ea56a8240d088be362c1402d7e6"
format_stamp: "Formatted at 2026-09-24 03:04:12 on dist-test-slave-wgsb"
server_key: "096863aaffb5a52a55836f0dc6f5df17"
server_key_iv: "cce97ea32eba3fff29504f4b33e8a09b"
server_key_version: "kuduclusterkey@0"
I20260924 03:04:12.642227 11589 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 03:04:12.663689 11604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:04:12.663960 11589 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 03:04:12.664088 11589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "0d225ea56a8240d088be362c1402d7e6"
format_stamp: "Formatted at 2026-09-24 03:04:12 on dist-test-slave-wgsb"
server_key: "096863aaffb5a52a55836f0dc6f5df17"
server_key_iv: "cce97ea32eba3fff29504f4b33e8a09b"
server_key_version: "kuduclusterkey@0"
I20260924 03:04:12.664213 11589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:04:12.676170 11589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:04:12.678642 11589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:04:12.678821 11589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:04:12.684420 11589 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33581
I20260924 03:04:12.684445 11656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33581 every 8 connection(s)
I20260924 03:04:12.684850 11589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:04:12.685600 11657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:04:12.687527 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 11589
I20260924 03:04:12.687676 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:04:12.687896 392 external_mini_cluster.cc:1428] Setting key 23424980d59f8f007fa94527ecdff53d
I20260924 03:04:12.689157 11657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d225ea56a8240d088be362c1402d7e6: Bootstrap starting.
I20260924 03:04:12.689982 11657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d225ea56a8240d088be362c1402d7e6: Neither blocks nor log segments found. Creating new log.
I20260924 03:04:12.690337 11657 log.cc:824] T 00000000000000000000000000000000 P 0d225ea56a8240d088be362c1402d7e6: Log is configured to *not* fsync() on all Append() calls
I20260924 03:04:12.691154 11657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d225ea56a8240d088be362c1402d7e6: No bootstrap required, opened a new log
I20260924 03:04:12.693682 11657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d225ea56a8240d088be362c1402d7e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d225ea56a8240d088be362c1402d7e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } }
I20260924 03:04:12.693892 11657 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d225ea56a8240d088be362c1402d7e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:04:12.693960 11657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d225ea56a8240d088be362c1402d7e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d225ea56a8240d088be362c1402d7e6, State: Initialized, Role: FOLLOWER
I20260924 03:04:12.694118 11657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d225ea56a8240d088be362c1402d7e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d225ea56a8240d088be362c1402d7e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } }
I20260924 03:04:12.694219 11657 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d225ea56a8240d088be362c1402d7e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:04:12.694257 11657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d225ea56a8240d088be362c1402d7e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:04:12.694336 11657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d225ea56a8240d088be362c1402d7e6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:04:12.695166 11657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d225ea56a8240d088be362c1402d7e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d225ea56a8240d088be362c1402d7e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } }
I20260924 03:04:12.695303 11657 leader_election.cc:302] T 00000000000000000000000000000000 P 0d225ea56a8240d088be362c1402d7e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d225ea56a8240d088be362c1402d7e6; no voters:
I20260924 03:04:12.695497 11657 leader_election.cc:288] T 00000000000000000000000000000000 P 0d225ea56a8240d088be362c1402d7e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:04:12.695559 11662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d225ea56a8240d088be362c1402d7e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:04:12.695879 11662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d225ea56a8240d088be362c1402d7e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d225ea56a8240d088be362c1402d7e6, State: Running, Role: LEADER
I20260924 03:04:12.695930 11657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d225ea56a8240d088be362c1402d7e6 [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 03:04:12 dist-test-slave-wgsb krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:04:12.696024 11662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d225ea56a8240d088be362c1402d7e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d225ea56a8240d088be362c1402d7e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } }
I20260924 03:04:12.697679 11663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d225ea56a8240d088be362c1402d7e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d225ea56a8240d088be362c1402d7e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d225ea56a8240d088be362c1402d7e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } } }
I20260924 03:04:12.697835 11663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d225ea56a8240d088be362c1402d7e6 [sys.catalog]: This master's current role is: LEADER
I20260924 03:04:12.698040 11663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d225ea56a8240d088be362c1402d7e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d225ea56a8240d088be362c1402d7e6. Latest consensus state: current_term: 1 leader_uuid: "0d225ea56a8240d088be362c1402d7e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d225ea56a8240d088be362c1402d7e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } } }
I20260924 03:04:12.698191 11663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d225ea56a8240d088be362c1402d7e6 [sys.catalog]: This master's current role is: LEADER
I20260924 03:04:12.698378 11672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:04:12.698446 11657 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 03:04:12.698987 11672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:04:12.701030 11660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:04:12.688467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56414 (local address 127.0.98.62:33581)
0924 03:04:12.688706 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 03:04:12.688717 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:04:12.688753 (+ 36us) server_negotiation.cc:407] Connection header received
0924 03:04:12.689298 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:04:12.689307 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:04:12.689372 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:04:12.689534 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:04:12.690438 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:04:12.691834 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:04:12.693098 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:04:12.693804 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:04:12.693923 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:04:12.697306 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:04:12.697310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:04:12.697310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:04:12.697323 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:04:12.699264 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:04:12.699717 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:04:12.699720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:04:12.699720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:04:12.699819 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:04:12.700077 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:04:12.700077 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:04:12.700078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:04:12.700405 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:04:12.700564 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 03:04:12.700704 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 03:04:12.700824 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":80,"threads_started":1}
I20260924 03:04:12.702621 11672 catalog_manager.cc:1347] Generated new cluster ID: 5e8c90b3e0ac4b24aa2912d59822c90e
I20260924 03:04:12.702692 11672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:04:12.712071 11679 mvcc.cc:205] Tried to move back new op lower bound from 7332737239863947264 to 7332737239846768640. Current Snapshot: MvccSnapshot[applied={T|T < 7332737239863947264 or (T in {7332737239863947264})}]
I20260924 03:04:12.737193 11672 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:04:12.738185 11672 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:04:12.755750 11672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d225ea56a8240d088be362c1402d7e6: Generated new TSK 0
I20260924 03:04:12.756093 11672 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:04:13.574851 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39849/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33581
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:04:13.699066 11702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:04:13.699353 11702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:04:13.699388 11702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:04:13.701087 11702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:04:13.701144 11702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:04:13.701184 11702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:04:13.703389 11702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:39849/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:04:13.703673 11702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:04:13.703995 11702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:04:13.710880 11708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:04:13.711319 11707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:04:13.711503 11710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:04:13.712841 11702 server_base.cc:1034] running on GCE node
I20260924 03:04:13.713052 11702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:04:13.713289 11702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:04:13.714445 11702 hybrid_clock.cc:648] HybridClock initialized: now 1790219053714422 us; error 41 us; skew 500 ppm
Sep 24 03:04:13 dist-test-slave-wgsb krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:04:13.717828 11702 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:04:13.718401 11702 webserver.cc:460] Webserver started at http://127.0.98.1:38233/ using document root <none> and password file <none>
I20260924 03:04:13.718703 11702 fs_manager.cc:359] Metadata directory not provided
I20260924 03:04:13.718784 11702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:04:13.718994 11702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:04:13 dist-test-slave-wgsb krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:04:13.733700 11702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-0/data/instance:
uuid: "2f5aae0f03f744509086788419a68aa7"
format_stamp: "Formatted at 2026-09-24 03:04:13 on dist-test-slave-wgsb"
server_key: "2f3eaae164bf3310c1885cd3db5d381e"
server_key_iv: "78bf4af606e71915e6136d0d20372e5d"
server_key_version: "kuduclusterkey@0"
I20260924 03:04:13.734263 11702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2f5aae0f03f744509086788419a68aa7"
format_stamp: "Formatted at 2026-09-24 03:04:13 on dist-test-slave-wgsb"
server_key: "2f3eaae164bf3310c1885cd3db5d381e"
server_key_iv: "78bf4af606e71915e6136d0d20372e5d"
server_key_version: "kuduclusterkey@0"
I20260924 03:04:13.737457 11702 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260924 03:04:13.749539 11717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:04:13.749974 11702 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:04:13.750067 11702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "2f5aae0f03f744509086788419a68aa7"
format_stamp: "Formatted at 2026-09-24 03:04:13 on dist-test-slave-wgsb"
server_key: "2f3eaae164bf3310c1885cd3db5d381e"
server_key_iv: "78bf4af606e71915e6136d0d20372e5d"
server_key_version: "kuduclusterkey@0"
I20260924 03:04:13.750169 11702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:04:13.773381 11702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:04:13.775873 11702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:04:13.776079 11702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:04:13.776324 11702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:04:13.776731 11702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:04:13.776818 11702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:04:13.776947 11702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:04:13.777016 11702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:04:13.784286 11702 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34927
I20260924 03:04:13.784795 11702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:04:13.789438 11832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34927 every 8 connection(s)
I20260924 03:04:13.792140 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 11702
I20260924 03:04:13.792250 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:04:13.792459 392 external_mini_cluster.cc:1428] Setting key 051480cb4e95193aeba276f9f1771234
Sep 24 03:04:13 dist-test-slave-wgsb krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:04:13.802268 11836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:04:13.790205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.1:34345)
0924 03:04:13.790564 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0924 03:04:13.790610 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 03:04:13.791214 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:04:13.791640 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:04:13.791645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:04:13.791902 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:04:13.792274 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:04:13.792329 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:04:13.794052 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:04:13.794053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:04:13.795349 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:04:13.795352 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:04:13.795906 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:04:13.795907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:04:13.796043 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:04:13.796629 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:04:13.796640 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:04:13.798367 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:04:13.800348 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:04:13.800356 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:04:13.800359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:04:13.800694 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:04:13.800976 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:04:13.800977 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:04:13.800977 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:04:13.801115 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:04:13.801441 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:04:13.801447 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:04:13.801650 (+ 203us) client_negotiation.cc:769] Sending connection context
0924 03:04:13.801753 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 03:04:13.802018 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":84,"threads_started":1}
I20260924 03:04:13.802681 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:04:13.790201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34345 (local address 127.0.98.62:33581)
0924 03:04:13.791405 (+ 1204us) server_negotiation.cc:206] Beginning negotiation
0924 03:04:13.791407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:04:13.791415 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:04:13.791443 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:04:13.791444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:04:13.791456 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:04:13.792415 (+ 959us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:04:13.792466 (+ 51us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:04:13.793937 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:04:13.795454 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:04:13.795791 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:04:13.798254 (+ 2463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:04:13.798472 (+ 218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:04:13.798474 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:04:13.798474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:04:13.798482 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:04:13.800230 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:04:13.800804 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:04:13.800805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:04:13.800806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:04:13.800887 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:04:13.801205 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:04:13.801205 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:04:13.801206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:04:13.801360 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:04:13.802422 (+ 1062us) server_negotiation.cc:1035] Waiting for connection context
0924 03:04:13.802466 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 03:04:13.802567 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1181,"thread_start_us":59,"threads_started":1}
I20260924 03:04:13.803195 11833 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33581
I20260924 03:04:13.803305 11833 heartbeater.cc:464] Registering TS with master...
I20260924 03:04:13.803500 11833 heartbeater.cc:511] Master 127.0.98.62:33581 requested a full tablet report, sending...
I20260924 03:04:13.804171 11621 ts_manager.cc:194] Registered new tserver with Master: 2f5aae0f03f744509086788419a68aa7 (127.0.98.1:34927)
I20260924 03:04:13.805047 11621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34345
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:04:13.836162 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39849/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33581
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:04:13.956444 11839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:04:13.956787 11839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:04:13.956869 11839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:04:13.958519 11839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:04:13.958727 11839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:04:13.958827 11839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:04:13.960963 11839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:39849/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:04:13.961476 11839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:04:13.961860 11839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:04:13.968264 11847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:04:13.968686 11846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:04:13.969046 11849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:04:13.970057 11839 server_base.cc:1034] running on GCE node
I20260924 03:04:13.970396 11839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:04:13.970657 11839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:04:13.971787 11839 hybrid_clock.cc:648] HybridClock initialized: now 1790219053971768 us; error 33 us; skew 500 ppm
Sep 24 03:04:13 dist-test-slave-wgsb krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:04:13.974969 11839 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:04:13.975442 11839 webserver.cc:460] Webserver started at http://127.0.98.2:32993/ using document root <none> and password file <none>
I20260924 03:04:13.975716 11839 fs_manager.cc:359] Metadata directory not provided
I20260924 03:04:13.975775 11839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:04:13.975893 11839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:04:13 dist-test-slave-wgsb krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:04:13.989092 11839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-1/data/instance:
uuid: "5a635b9e9743405fbe03088bae61a13c"
format_stamp: "Formatted at 2026-09-24 03:04:13 on dist-test-slave-wgsb"
server_key: "92f87b7a086eb6b97020cd4df6f97ed2"
server_key_iv: "9e9426c7fd5c4a7a65b01065b1732e42"
server_key_version: "kuduclusterkey@0"
I20260924 03:04:13.989653 11839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5a635b9e9743405fbe03088bae61a13c"
format_stamp: "Formatted at 2026-09-24 03:04:13 on dist-test-slave-wgsb"
server_key: "92f87b7a086eb6b97020cd4df6f97ed2"
server_key_iv: "9e9426c7fd5c4a7a65b01065b1732e42"
server_key_version: "kuduclusterkey@0"
I20260924 03:04:13.992002 11839 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
Sep 24 03:04:13 dist-test-slave-wgsb krb5kdc[11145](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790219053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:04:14.011124 11856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:04:14.012385 11839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 03:04:14.012487 11839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "5a635b9e9743405fbe03088bae61a13c"
format_stamp: "Formatted at 2026-09-24 03:04:13 on dist-test-slave-wgsb"
server_key: "92f87b7a086eb6b97020cd4df6f97ed2"
server_key_iv: "9e9426c7fd5c4a7a65b01065b1732e42"
server_key_version: "kuduclusterkey@0"
I20260924 03:04:14.012584 11839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:04:14.046185 11839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:04:14.049011 11839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:04:14.049194 11839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:04:14.049419 11839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:04:14.049786 11839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:04:14.049870 11839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:04:14.049937 11839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:04:14.049985 11839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:04:14.057242 11839 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46825
I20260924 03:04:14.058075 11839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:04:14.060472 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 11839
I20260924 03:04:14.060588 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:04:14.060789 392 external_mini_cluster.cc:1428] Setting key b8d2515022449c935a0ae767dcd354f8
I20260924 03:04:14.060762 11969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46825 every 8 connection(s)
Sep 24 03:04:14 dist-test-slave-wgsb krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 03:04:14.073730 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:04:14.061835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55627 (local address 127.0.98.62:33581)
0924 03:04:14.061893 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 03:04:14.061896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:04:14.062065 (+ 169us) server_negotiation.cc:407] Connection header received
0924 03:04:14.062830 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:04:14.062831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:04:14.062844 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:04:14.062879 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:04:14.064282 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.065062 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.066621 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.066979 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.067024 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:04:14.070146 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:04:14.070148 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:04:14.070149 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:04:14.070157 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:04:14.072164 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:04:14.072650 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:04:14.072651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:04:14.072651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:04:14.072702 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:04:14.072969 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:04:14.072970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:04:14.072970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:04:14.073105 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:04:14.073143 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 03:04:14.073508 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 03:04:14.073610 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 03:04:14.074347 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:04:14.061586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.2:55627)
0924 03:04:14.061937 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0924 03:04:14.061984 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 03:04:14.062673 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:04:14.063078 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:04:14.063082 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:04:14.063849 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:04:14.064154 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:04:14.064161 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.065193 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.065194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:04:14.066514 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:04:14.066517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.067118 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.067119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:04:14.067269 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:04:14.067969 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:04:14.067980 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:04:14.070014 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:04:14.072266 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:04:14.072276 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:04:14.072278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:04:14.072570 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:04:14.072781 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:04:14.072781 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:04:14.072782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:04:14.072902 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:04:14.073183 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:04:14.073187 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:04:14.073369 (+ 182us) client_negotiation.cc:769] Sending connection context
0924 03:04:14.073941 (+ 572us) client_negotiation.cc:240] Negotiation successful
0924 03:04:14.074110 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":90,"threads_started":1}
I20260924 03:04:14.074831 11970 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33581
I20260924 03:04:14.074939 11970 heartbeater.cc:464] Registering TS with master...
I20260924 03:04:14.075145 11970 heartbeater.cc:511] Master 127.0.98.62:33581 requested a full tablet report, sending...
I20260924 03:04:14.075632 11621 ts_manager.cc:194] Registered new tserver with Master: 5a635b9e9743405fbe03088bae61a13c (127.0.98.2:46825)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 03:04:14.076148 11621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:04:14.096051 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39849/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33581
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:04:14.217039 11975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:04:14.217408 11975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:04:14.217468 11975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:04:14.219225 11975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:04:14.219296 11975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:04:14.219341 11975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:04:14.222090 11975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:39849/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:04:14.222384 11975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:04:14.222718 11975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:04:14.230517 11984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:04:14.230517 11982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:04:14.230885 11981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:04:14.232216 11975 server_base.cc:1034] running on GCE node
I20260924 03:04:14.232568 11975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:04:14.232892 11975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:04:14.236177 11975 hybrid_clock.cc:648] HybridClock initialized: now 1790219054236149 us; error 49 us; skew 500 ppm
Sep 24 03:04:14 dist-test-slave-wgsb krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:04:14.246286 11975 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:04:14.246955 11975 webserver.cc:460] Webserver started at http://127.0.98.3:35065/ using document root <none> and password file <none>
I20260924 03:04:14.247284 11975 fs_manager.cc:359] Metadata directory not provided
I20260924 03:04:14.247449 11975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:04:14.247685 11975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:04:14 dist-test-slave-wgsb krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:04:14.265913 11975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-2/data/instance:
uuid: "cd6d181458e94776b5c1a08bcb3eb094"
format_stamp: "Formatted at 2026-09-24 03:04:14 on dist-test-slave-wgsb"
server_key: "12620455d0d42dc8ab09f6181254fbff"
server_key_iv: "5e64fffac594e36d93d63d6955cafc9d"
server_key_version: "kuduclusterkey@0"
I20260924 03:04:14.266402 11975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cd6d181458e94776b5c1a08bcb3eb094"
format_stamp: "Formatted at 2026-09-24 03:04:14 on dist-test-slave-wgsb"
server_key: "12620455d0d42dc8ab09f6181254fbff"
server_key_iv: "5e64fffac594e36d93d63d6955cafc9d"
server_key_version: "kuduclusterkey@0"
I20260924 03:04:14.280043 11975 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.000s sys 0.004s
I20260924 03:04:14.295128 11991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:04:14.295914 11975 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:04:14.296083 11975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "cd6d181458e94776b5c1a08bcb3eb094"
format_stamp: "Formatted at 2026-09-24 03:04:14 on dist-test-slave-wgsb"
server_key: "12620455d0d42dc8ab09f6181254fbff"
server_key_iv: "5e64fffac594e36d93d63d6955cafc9d"
server_key_version: "kuduclusterkey@0"
I20260924 03:04:14.296259 11975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:04:14.321324 11975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:04:14.323673 11975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:04:14.323892 11975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:04:14.324201 11975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:04:14.324589 11975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:04:14.324647 11975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:04:14.324731 11975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:04:14.324762 11975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:04:14.332567 11975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43693
I20260924 03:04:14.332952 11975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:04:14.333806 12104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43693 every 8 connection(s)
Sep 24 03:04:14 dist-test-slave-wgsb krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:04:14.341218 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 11975
I20260924 03:04:14.341315 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:04:14.341805 392 external_mini_cluster.cc:1428] Setting key 38482e7ffafe07e28123dc32387ed1d5
I20260924 03:04:14.344596 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:04:14.333728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60087 (local address 127.0.98.62:33581)
0924 03:04:14.333776 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 03:04:14.333778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:04:14.334142 (+ 364us) server_negotiation.cc:407] Connection header received
0924 03:04:14.334731 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:04:14.334732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:04:14.334744 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:04:14.334793 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:04:14.335351 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.336048 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.337502 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.337786 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.337834 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:04:14.340534 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:04:14.340536 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:04:14.340536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:04:14.340543 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:04:14.342300 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:04:14.343115 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:04:14.343116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:04:14.343117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:04:14.343173 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:04:14.343467 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:04:14.343468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:04:14.343468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:04:14.343622 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:04:14.343672 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 03:04:14.344418 (+ 746us) server_negotiation.cc:299] Negotiation successful
0924 03:04:14.344502 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:04:14.345345 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:04:14.333732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.3:60087)
0924 03:04:14.333992 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 03:04:14.334031 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 03:04:14.334620 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:04:14.334860 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:04:14.334864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:04:14.335057 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:04:14.335251 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:04:14.335255 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.336157 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.336158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:04:14.337393 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:04:14.337397 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.337934 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.337935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:04:14.338075 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:04:14.338696 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:04:14.338707 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:04:14.340421 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:04:14.342719 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:04:14.342726 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:04:14.342728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:04:14.343022 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:04:14.343245 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:04:14.343246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:04:14.343246 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:04:14.343388 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:04:14.344053 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:04:14.344059 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:04:14.344235 (+ 176us) client_negotiation.cc:769] Sending connection context
0924 03:04:14.344924 (+ 689us) client_negotiation.cc:240] Negotiation successful
0924 03:04:14.345093 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":91,"threads_started":1}
I20260924 03:04:14.345989 12105 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33581
I20260924 03:04:14.346091 12105 heartbeater.cc:464] Registering TS with master...
I20260924 03:04:14.346276 12105 heartbeater.cc:511] Master 127.0.98.62:33581 requested a full tablet report, sending...
I20260924 03:04:14.346666 11621 ts_manager.cc:194] Registered new tserver with Master: cd6d181458e94776b5c1a08bcb3eb094 (127.0.98.3:43693)
I20260924 03:04:14.347142 11621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60087
I20260924 03:04:14.357419 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:04:14.377802 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:04:14.367256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56422 (local address 127.0.98.62:33581)
0924 03:04:14.367355 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 03:04:14.367358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:04:14.367787 (+ 429us) server_negotiation.cc:407] Connection header received
0924 03:04:14.367919 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:04:14.367920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:04:14.367934 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:04:14.367970 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:04:14.368929 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.369696 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.371809 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.372098 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.372146 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:04:14.374084 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:04:14.374086 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:04:14.374086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:04:14.374096 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:04:14.376209 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:04:14.376723 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:04:14.376725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:04:14.376725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:04:14.376791 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:04:14.377092 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:04:14.377093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:04:14.377094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:04:14.377243 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:04:14.377327 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 03:04:14.377594 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 03:04:14.377695 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 03:04:14.380409 11621 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:04:14.382985 11621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:04:14.406032 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:04:14.395069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34927 (local address 127.0.0.1:53224)
0924 03:04:14.396673 (+ 1604us) negotiation.cc:107] Waiting for socket to connect
0924 03:04:14.396723 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 03:04:14.396794 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:04:14.397378 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:04:14.397382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:04:14.397384 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:04:14.397524 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:04:14.397526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.399771 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.399772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:04:14.401616 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:04:14.401619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.405684 (+ 4065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.405685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:04:14.405794 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:04:14.405809 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 03:04:14.405881 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 03:04:14.405939 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1569,"thread_start_us":90,"threads_started":1}
I20260924 03:04:14.408260 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:04:14.395378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43693 (local address 127.0.0.1:59156)
0924 03:04:14.396906 (+ 1528us) negotiation.cc:107] Waiting for socket to connect
0924 03:04:14.396915 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:04:14.397023 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:04:14.398756 (+ 1733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:04:14.398757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:04:14.398759 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:04:14.398816 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:04:14.398818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.402974 (+ 4156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.402975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:04:14.404569 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:04:14.404572 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.407991 (+ 3419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.407992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:04:14.408063 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:04:14.408067 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:04:14.408116 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 03:04:14.408162 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1512,"thread_start_us":76,"threads_started":1}
I20260924 03:04:14.408806 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:04:14.395748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53224 (local address 127.0.98.1:34927)
0924 03:04:14.397132 (+ 1384us) server_negotiation.cc:206] Beginning negotiation
0924 03:04:14.397138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:04:14.397148 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:04:14.397185 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:04:14.397187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:04:14.397204 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:04:14.397293 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:04:14.397611 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.399678 (+ 2067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.404687 (+ 5009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.405504 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.405580 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:04:14.405647 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 03:04:14.408668 (+ 3021us) server_negotiation.cc:299] Negotiation successful
0924 03:04:14.408731 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1348,"thread_start_us":52,"threads_started":1}
I20260924 03:04:14.409708 11756 tablet_service.cc:1467] Processing CreateTablet for tablet 978490eaa7a44a20876f87e8aec72cd5 (DEFAULT_TABLE table=test-table [id=deafb7cbfa594fc6b1d0ffabc989bd24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:04:14.409984 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:04:14.396204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46825 (local address 127.0.0.1:37840)
0924 03:04:14.404334 (+ 8130us) negotiation.cc:107] Waiting for socket to connect
0924 03:04:14.404347 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:04:14.404409 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:04:14.404796 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:04:14.404797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:04:14.404799 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:04:14.404846 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:04:14.404847 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.406125 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.406126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:04:14.407266 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:04:14.407268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.409733 (+ 2465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.409734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:04:14.409797 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:04:14.409801 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:04:14.409852 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 03:04:14.409894 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8104,"thread_start_us":81,"threads_started":1}
I20260924 03:04:14.410090 11756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 978490eaa7a44a20876f87e8aec72cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:04:14.412197 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:04:14.396331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37840 (local address 127.0.98.2:46825)
0924 03:04:14.404549 (+ 8218us) server_negotiation.cc:206] Beginning negotiation
0924 03:04:14.404555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:04:14.404565 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:04:14.404604 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:04:14.404606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:04:14.404622 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:04:14.404700 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:04:14.404929 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.406027 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.408772 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.409547 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.409622 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:04:14.409696 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 03:04:14.412022 (+ 2326us) server_negotiation.cc:299] Negotiation successful
0924 03:04:14.412091 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8176,"thread_start_us":85,"threads_started":1}
I20260924 03:04:14.413022 11904 tablet_service.cc:1467] Processing CreateTablet for tablet 978490eaa7a44a20876f87e8aec72cd5 (DEFAULT_TABLE table=test-table [id=deafb7cbfa594fc6b1d0ffabc989bd24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:04:14.413359 11904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 978490eaa7a44a20876f87e8aec72cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:04:14.414419 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:04:14.395891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59156 (local address 127.0.98.3:43693)
0924 03:04:14.398532 (+ 2641us) server_negotiation.cc:206] Beginning negotiation
0924 03:04:14.398538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:04:14.398548 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:04:14.398587 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:04:14.398589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:04:14.398605 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:04:14.398677 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:04:14.401740 (+ 3063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.402874 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.406633 (+ 3759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.407812 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.407886 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:04:14.407957 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:04:14.414270 (+ 6313us) server_negotiation.cc:299] Negotiation successful
0924 03:04:14.414333 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2229,"thread_start_us":89,"threads_started":1}
I20260924 03:04:14.415040 12039 tablet_service.cc:1467] Processing CreateTablet for tablet 978490eaa7a44a20876f87e8aec72cd5 (DEFAULT_TABLE table=test-table [id=deafb7cbfa594fc6b1d0ffabc989bd24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:04:14.415378 12039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 978490eaa7a44a20876f87e8aec72cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:04:14.419029 12126 tablet_bootstrap.cc:492] T 978490eaa7a44a20876f87e8aec72cd5 P cd6d181458e94776b5c1a08bcb3eb094: Bootstrap starting.
I20260924 03:04:14.420222 12126 tablet_bootstrap.cc:654] T 978490eaa7a44a20876f87e8aec72cd5 P cd6d181458e94776b5c1a08bcb3eb094: Neither blocks nor log segments found. Creating new log.
I20260924 03:04:14.420634 12126 log.cc:824] T 978490eaa7a44a20876f87e8aec72cd5 P cd6d181458e94776b5c1a08bcb3eb094: Log is configured to *not* fsync() on all Append() calls
I20260924 03:04:14.421489 12126 tablet_bootstrap.cc:492] T 978490eaa7a44a20876f87e8aec72cd5 P cd6d181458e94776b5c1a08bcb3eb094: No bootstrap required, opened a new log
I20260924 03:04:14.421558 12126 ts_tablet_manager.cc:1397] T 978490eaa7a44a20876f87e8aec72cd5 P cd6d181458e94776b5c1a08bcb3eb094: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 03:04:14.423905 12125 tablet_bootstrap.cc:492] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c: Bootstrap starting.
I20260924 03:04:14.424026 12126 raft_consensus.cc:348] T 978490eaa7a44a20876f87e8aec72cd5 P cd6d181458e94776b5c1a08bcb3eb094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a635b9e9743405fbe03088bae61a13c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46825 } } peers { permanent_uuid: "2f5aae0f03f744509086788419a68aa7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34927 } } peers { permanent_uuid: "cd6d181458e94776b5c1a08bcb3eb094" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43693 } }
I20260924 03:04:14.424203 12126 raft_consensus.cc:374] T 978490eaa7a44a20876f87e8aec72cd5 P cd6d181458e94776b5c1a08bcb3eb094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:04:14.424254 12126 raft_consensus.cc:729] T 978490eaa7a44a20876f87e8aec72cd5 P cd6d181458e94776b5c1a08bcb3eb094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd6d181458e94776b5c1a08bcb3eb094, State: Initialized, Role: FOLLOWER
I20260924 03:04:14.424798 12126 consensus_queue.cc:260] T 978490eaa7a44a20876f87e8aec72cd5 P cd6d181458e94776b5c1a08bcb3eb094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a635b9e9743405fbe03088bae61a13c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46825 } } peers { permanent_uuid: "2f5aae0f03f744509086788419a68aa7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34927 } } peers { permanent_uuid: "cd6d181458e94776b5c1a08bcb3eb094" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43693 } }
I20260924 03:04:14.424989 12125 tablet_bootstrap.cc:654] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c: Neither blocks nor log segments found. Creating new log.
I20260924 03:04:14.425256 12126 ts_tablet_manager.cc:1428] T 978490eaa7a44a20876f87e8aec72cd5 P cd6d181458e94776b5c1a08bcb3eb094: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260924 03:04:14.425369 12124 tablet_bootstrap.cc:492] T 978490eaa7a44a20876f87e8aec72cd5 P 2f5aae0f03f744509086788419a68aa7: Bootstrap starting.
I20260924 03:04:14.425434 12105 heartbeater.cc:503] Master 127.0.98.62:33581 was elected leader, sending a full tablet report...
I20260924 03:04:14.426025 12125 log.cc:824] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:04:14.426925 12124 tablet_bootstrap.cc:654] T 978490eaa7a44a20876f87e8aec72cd5 P 2f5aae0f03f744509086788419a68aa7: Neither blocks nor log segments found. Creating new log.
I20260924 03:04:14.427327 12124 log.cc:824] T 978490eaa7a44a20876f87e8aec72cd5 P 2f5aae0f03f744509086788419a68aa7: Log is configured to *not* fsync() on all Append() calls
I20260924 03:04:14.428284 12125 tablet_bootstrap.cc:492] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c: No bootstrap required, opened a new log
I20260924 03:04:14.428421 12125 ts_tablet_manager.cc:1397] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 03:04:14.430601 12125 raft_consensus.cc:348] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a635b9e9743405fbe03088bae61a13c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46825 } } peers { permanent_uuid: "2f5aae0f03f744509086788419a68aa7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34927 } } peers { permanent_uuid: "cd6d181458e94776b5c1a08bcb3eb094" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43693 } }
I20260924 03:04:14.431629 12124 tablet_bootstrap.cc:492] T 978490eaa7a44a20876f87e8aec72cd5 P 2f5aae0f03f744509086788419a68aa7: No bootstrap required, opened a new log
I20260924 03:04:14.432224 12124 ts_tablet_manager.cc:1397] T 978490eaa7a44a20876f87e8aec72cd5 P 2f5aae0f03f744509086788419a68aa7: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260924 03:04:14.435087 12124 raft_consensus.cc:348] T 978490eaa7a44a20876f87e8aec72cd5 P 2f5aae0f03f744509086788419a68aa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a635b9e9743405fbe03088bae61a13c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46825 } } peers { permanent_uuid: "2f5aae0f03f744509086788419a68aa7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34927 } } peers { permanent_uuid: "cd6d181458e94776b5c1a08bcb3eb094" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43693 } }
I20260924 03:04:14.435377 12124 raft_consensus.cc:374] T 978490eaa7a44a20876f87e8aec72cd5 P 2f5aae0f03f744509086788419a68aa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:04:14.435472 12124 raft_consensus.cc:729] T 978490eaa7a44a20876f87e8aec72cd5 P 2f5aae0f03f744509086788419a68aa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f5aae0f03f744509086788419a68aa7, State: Initialized, Role: FOLLOWER
I20260924 03:04:14.435740 12124 consensus_queue.cc:260] T 978490eaa7a44a20876f87e8aec72cd5 P 2f5aae0f03f744509086788419a68aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a635b9e9743405fbe03088bae61a13c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46825 } } peers { permanent_uuid: "2f5aae0f03f744509086788419a68aa7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34927 } } peers { permanent_uuid: "cd6d181458e94776b5c1a08bcb3eb094" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43693 } }
I20260924 03:04:14.436139 12124 ts_tablet_manager.cc:1428] T 978490eaa7a44a20876f87e8aec72cd5 P 2f5aae0f03f744509086788419a68aa7: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260924 03:04:14.436326 11833 heartbeater.cc:503] Master 127.0.98.62:33581 was elected leader, sending a full tablet report...
I20260924 03:04:14.437233 12125 raft_consensus.cc:374] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:04:14.437384 12125 raft_consensus.cc:729] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a635b9e9743405fbe03088bae61a13c, State: Initialized, Role: FOLLOWER
I20260924 03:04:14.437602 12125 consensus_queue.cc:260] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a635b9e9743405fbe03088bae61a13c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46825 } } peers { permanent_uuid: "2f5aae0f03f744509086788419a68aa7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34927 } } peers { permanent_uuid: "cd6d181458e94776b5c1a08bcb3eb094" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43693 } }
I20260924 03:04:14.438010 12125 ts_tablet_manager.cc:1428] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20260924 03:04:14.438596 11970 heartbeater.cc:503] Master 127.0.98.62:33581 was elected leader, sending a full tablet report...
Sep 24 03:04:14 dist-test-slave-wgsb krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260924 03:04:14.541040 11834 tablet.cc:2369] T 978490eaa7a44a20876f87e8aec72cd5 P 2f5aae0f03f744509086788419a68aa7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:04:14.561657 11971 tablet.cc:2369] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:04:14.583763 12106 tablet.cc:2369] T 978490eaa7a44a20876f87e8aec72cd5 P cd6d181458e94776b5c1a08bcb3eb094: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:04:14.831799 12132 raft_consensus.cc:482] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:04:14.832005 12132 raft_consensus.cc:504] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a635b9e9743405fbe03088bae61a13c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46825 } } peers { permanent_uuid: "2f5aae0f03f744509086788419a68aa7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34927 } } peers { permanent_uuid: "cd6d181458e94776b5c1a08bcb3eb094" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43693 } }
I20260924 03:04:14.832394 12132 leader_election.cc:288] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f5aae0f03f744509086788419a68aa7 (127.0.98.1:34927), cd6d181458e94776b5c1a08bcb3eb094 (127.0.98.3:43693)
I20260924 03:04:14.840684 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:04:14.833202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53221 (local address 127.0.98.1:34927)
0924 03:04:14.833373 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 03:04:14.833376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:04:14.836515 (+ 3139us) server_negotiation.cc:407] Connection header received
0924 03:04:14.836635 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:04:14.836637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:04:14.836651 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:04:14.836681 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:04:14.836906 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.837576 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.839686 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.840278 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.840325 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:04:14.840337 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 03:04:14.840550 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 03:04:14.840604 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260924 03:04:14.840945 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:04:14.832704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34927 (local address 127.0.98.2:53221)
0924 03:04:14.836401 (+ 3697us) negotiation.cc:107] Waiting for socket to connect
0924 03:04:14.836418 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:04:14.836553 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:04:14.836761 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:04:14.836762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:04:14.836764 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:04:14.836833 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:04:14.836835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.837694 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.837695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:04:14.839553 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:04:14.839555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.840398 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.840399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:04:14.840469 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:04:14.840473 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:04:14.840796 (+ 323us) client_negotiation.cc:240] Negotiation successful
0924 03:04:14.840856 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3674,"thread_start_us":90,"threads_started":1}
I20260924 03:04:14.841377 11787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "978490eaa7a44a20876f87e8aec72cd5" candidate_uuid: "5a635b9e9743405fbe03088bae61a13c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f5aae0f03f744509086788419a68aa7" is_pre_election: true
I20260924 03:04:14.841562 11787 raft_consensus.cc:2454] T 978490eaa7a44a20876f87e8aec72cd5 P 2f5aae0f03f744509086788419a68aa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a635b9e9743405fbe03088bae61a13c in term 0.
I20260924 03:04:14.841820 11859 leader_election.cc:302] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f5aae0f03f744509086788419a68aa7, 5a635b9e9743405fbe03088bae61a13c; no voters:
I20260924 03:04:14.841943 12132 raft_consensus.cc:2781] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:04:14.842002 12132 raft_consensus.cc:482] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:04:14.842026 12132 raft_consensus.cc:3037] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:04:14.842780 12132 raft_consensus.cc:504] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a635b9e9743405fbe03088bae61a13c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46825 } } peers { permanent_uuid: "2f5aae0f03f744509086788419a68aa7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34927 } } peers { permanent_uuid: "cd6d181458e94776b5c1a08bcb3eb094" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43693 } }
I20260924 03:04:14.842942 12132 leader_election.cc:288] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c [CANDIDATE]: Term 1 election: Requested vote from peers 2f5aae0f03f744509086788419a68aa7 (127.0.98.1:34927), cd6d181458e94776b5c1a08bcb3eb094 (127.0.98.3:43693)
I20260924 03:04:14.843081 11787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "978490eaa7a44a20876f87e8aec72cd5" candidate_uuid: "5a635b9e9743405fbe03088bae61a13c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f5aae0f03f744509086788419a68aa7"
I20260924 03:04:14.843189 11787 raft_consensus.cc:3037] T 978490eaa7a44a20876f87e8aec72cd5 P 2f5aae0f03f744509086788419a68aa7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:04:14.844077 11787 raft_consensus.cc:2454] T 978490eaa7a44a20876f87e8aec72cd5 P 2f5aae0f03f744509086788419a68aa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a635b9e9743405fbe03088bae61a13c in term 1.
I20260924 03:04:14.844403 11859 leader_election.cc:302] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f5aae0f03f744509086788419a68aa7, 5a635b9e9743405fbe03088bae61a13c; no voters:
I20260924 03:04:14.844527 12132 raft_consensus.cc:2781] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:04:14.844681 12132 raft_consensus.cc:686] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c [term 1 LEADER]: Becoming Leader. State: Replica: 5a635b9e9743405fbe03088bae61a13c, State: Running, Role: LEADER
I20260924 03:04:14.844786 12132 consensus_queue.cc:237] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a635b9e9743405fbe03088bae61a13c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46825 } } peers { permanent_uuid: "2f5aae0f03f744509086788419a68aa7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34927 } } peers { permanent_uuid: "cd6d181458e94776b5c1a08bcb3eb094" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43693 } }
I20260924 03:04:14.846153 11621 catalog_manager.cc:5495] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a635b9e9743405fbe03088bae61a13c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "5a635b9e9743405fbe03088bae61a13c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a635b9e9743405fbe03088bae61a13c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f5aae0f03f744509086788419a68aa7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd6d181458e94776b5c1a08bcb3eb094" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43693 } health_report { overall_health: UNKNOWN } } }
I20260924 03:04:14.846442 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:04:14.833403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39025 (local address 127.0.98.3:43693)
0924 03:04:14.833448 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 03:04:14.833449 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:04:14.839471 (+ 6022us) server_negotiation.cc:407] Connection header received
0924 03:04:14.839554 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:04:14.839555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:04:14.839567 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:04:14.839615 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:04:14.839839 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.840501 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.845440 (+ 4939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.845873 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.845981 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:04:14.845993 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 03:04:14.846325 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 03:04:14.846377 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:04:14.846652 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:04:14.832936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43693 (local address 127.0.98.2:39025)
0924 03:04:14.839402 (+ 6466us) negotiation.cc:107] Waiting for socket to connect
0924 03:04:14.839414 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:04:14.839476 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:04:14.839684 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:04:14.839685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:04:14.839686 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:04:14.839761 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:04:14.839762 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.840608 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.840609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:04:14.845259 (+ 4650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:04:14.845264 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.845985 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.845985 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:04:14.846186 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:04:14.846194 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 03:04:14.846513 (+ 319us) client_negotiation.cc:240] Negotiation successful
0924 03:04:14.846569 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6448,"thread_start_us":36,"threads_started":1}
I20260924 03:04:14.847044 12059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "978490eaa7a44a20876f87e8aec72cd5" candidate_uuid: "5a635b9e9743405fbe03088bae61a13c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd6d181458e94776b5c1a08bcb3eb094" is_pre_election: true
I20260924 03:04:14.847203 12059 raft_consensus.cc:2454] T 978490eaa7a44a20876f87e8aec72cd5 P cd6d181458e94776b5c1a08bcb3eb094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a635b9e9743405fbe03088bae61a13c in term 0.
I20260924 03:04:14.847354 12058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "978490eaa7a44a20876f87e8aec72cd5" candidate_uuid: "5a635b9e9743405fbe03088bae61a13c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd6d181458e94776b5c1a08bcb3eb094"
I20260924 03:04:14.847471 12058 raft_consensus.cc:3037] T 978490eaa7a44a20876f87e8aec72cd5 P cd6d181458e94776b5c1a08bcb3eb094 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:04:14.848609 12058 raft_consensus.cc:2454] T 978490eaa7a44a20876f87e8aec72cd5 P cd6d181458e94776b5c1a08bcb3eb094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a635b9e9743405fbe03088bae61a13c in term 1.
Sep 24 03:04:14 dist-test-slave-wgsb krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:04:14.957294 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:04:14.950722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37846 (local address 127.0.98.2:46825)
0924 03:04:14.950863 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 03:04:14.950866 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:04:14.951063 (+ 197us) server_negotiation.cc:407] Connection header received
0924 03:04:14.951112 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:04:14.951113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:04:14.951138 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:04:14.951218 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:04:14.951413 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.952173 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.955647 (+ 3474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:04:14.956136 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:04:14.956225 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:04:14.956482 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:04:14.956644 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:04:14.956847 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 03:04:14.957157 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 03:04:14.957191 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260924 03:04:14.964891 12058 raft_consensus.cc:1261] T 978490eaa7a44a20876f87e8aec72cd5 P cd6d181458e94776b5c1a08bcb3eb094 [term 1 FOLLOWER]: Refusing update from remote peer 5a635b9e9743405fbe03088bae61a13c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:04:14.965271 11787 raft_consensus.cc:1261] T 978490eaa7a44a20876f87e8aec72cd5 P 2f5aae0f03f744509086788419a68aa7 [term 1 FOLLOWER]: Refusing update from remote peer 5a635b9e9743405fbe03088bae61a13c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:04:14.966214 12135 consensus_queue.cc:1035] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f5aae0f03f744509086788419a68aa7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:04:14.966521 12132 consensus_queue.cc:1035] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd6d181458e94776b5c1a08bcb3eb094" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:04:14.978307 12144 mvcc.cc:205] Tried to move back new op lower bound from 7332737249132109824 to 7332737248644902912. Current Snapshot: MvccSnapshot[applied={T|T < 7332737249132109824}]
I20260924 03:04:14.979140 12147 mvcc.cc:205] Tried to move back new op lower bound from 7332737249132109824 to 7332737248644902912. Current Snapshot: MvccSnapshot[applied={T|T < 7332737249132109824}]
I20260924 03:04:14.984158 12142 mvcc.cc:205] Tried to move back new op lower bound from 7332737249132109824 to 7332737248644902912. Current Snapshot: MvccSnapshot[applied={T|T < 7332737249132109824}]
I20260924 03:04:14.987900 11621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:04:14.988055 11621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:04:14.988948 11621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d225ea56a8240d088be362c1402d7e6: Sending DeleteTablet for 3 replicas of tablet 978490eaa7a44a20876f87e8aec72cd5
I20260924 03:04:14.989730 12039 tablet_service.cc:1514] Processing DeleteTablet for tablet 978490eaa7a44a20876f87e8aec72cd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:04:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59156
I20260924 03:04:14.989899 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 11702
I20260924 03:04:14.989987 11904 tablet_service.cc:1514] Processing DeleteTablet for tablet 978490eaa7a44a20876f87e8aec72cd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:04:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37840
I20260924 03:04:14.999742 12153 tablet_replica.cc:330] T 978490eaa7a44a20876f87e8aec72cd5 P cd6d181458e94776b5c1a08bcb3eb094: stopping tablet replica
I20260924 03:04:15.000026 12154 tablet_replica.cc:330] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c: stopping tablet replica
I20260924 03:04:15.000269 12154 raft_consensus.cc:2229] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:04:15.000872 12153 raft_consensus.cc:2229] T 978490eaa7a44a20876f87e8aec72cd5 P cd6d181458e94776b5c1a08bcb3eb094 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:04:15.001374 12154 raft_consensus.cc:2258] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:04:15.001726 12153 raft_consensus.cc:2258] T 978490eaa7a44a20876f87e8aec72cd5 P cd6d181458e94776b5c1a08bcb3eb094 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:04:15.002790 12154 ts_tablet_manager.cc:1905] T 978490eaa7a44a20876f87e8aec72cd5 P 5a635b9e9743405fbe03088bae61a13c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:04:15.003031 12153 ts_tablet_manager.cc:1905] T 978490eaa7a44a20876f87e8aec72cd5 P cd6d181458e94776b5c1a08bcb3eb094: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 03:04:15.004052 11607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34927 (error 108)
I20260924 03:04:15.004266 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 11839
I20260924 03:04:15.007746 12153 ts_tablet_manager.cc:1918] T 978490eaa7a44a20876f87e8aec72cd5 P cd6d181458e94776b5c1a08bcb3eb094: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:04:15.007874 12153 log.cc:1196] T 978490eaa7a44a20876f87e8aec72cd5 P cd6d181458e94776b5c1a08bcb3eb094: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218948286999-392-0/minicluster-data/ts-2/wal/wals/978490eaa7a44a20876f87e8aec72cd5
I20260924 03:04:15.008181 12153 ts_tablet_manager.cc:1939] T 978490eaa7a44a20876f87e8aec72cd5 P cd6d181458e94776b5c1a08bcb3eb094: Deleting consensus metadata
I20260924 03:04:15.009265 11605 catalog_manager.cc:4841] TS cd6d181458e94776b5c1a08bcb3eb094 (127.0.98.3:43693): tablet 978490eaa7a44a20876f87e8aec72cd5 (table test-table [id=deafb7cbfa594fc6b1d0ffabc989bd24]) successfully deleted
I20260924 03:04:15.015697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 11975
I20260924 03:04:15.016424 12156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:04:15.007535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34927 (local address 127.0.0.1:53240)
0924 03:04:15.016296 (+ 8761us) negotiation.cc:107] Waiting for socket to connect
0924 03:04:15.016352 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7459,"thread_start_us":90,"threads_started":1}
W20260924 03:04:15.016589 11607 catalog_manager.cc:4584] TS 2f5aae0f03f744509086788419a68aa7 (127.0.98.1:34927): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 978490eaa7a44a20876f87e8aec72cd5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34927: connect: Connection refused (error 111)
I20260924 03:04:15.017014 12157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:04:15.014776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46825 (local address 127.0.0.1:37862)
0924 03:04:15.015743 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0924 03:04:15.016956 (+ 1213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":931,"thread_start_us":112,"threads_started":1}
I20260924 03:04:15.029271 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 11589
I20260924 03:04:15.038009 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 03:04:15.384147 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 03:04:15.384253 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 03:04:15.729941 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 03:04:15.730 UTC [11175] LOG: received smart shutdown request
2026-09-24 03:04:15.734 UTC [11175] LOG: background worker "logical replication launcher" (PID 11184) exited with exit code 1
2026-09-24 03:04:15.735 UTC [11179] LOG: shutting down
2026-09-24 03:04:16.000 UTC [11175] LOG: database system is shut down
2026-09-24T03:04:16Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48400 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:04:16 dist-test-slave-wgsb krb5kdc[12166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:04:16 dist-test-slave-wgsb krb5kdc[12166](info): set up 2 sockets
Sep 24 03:04:16 dist-test-slave-wgsb krb5kdc[12166](info): commencing operation
krb5kdc: starting...
W20260924 03:04:17.232554 392 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:04:17 dist-test-slave-wgsb krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:04:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:04:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 03:04:17.403443 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskptPCbv/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/postgres -l logfile start
2026-09-24 03:04:20.576 UTC [12196] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 03:04:20.576 UTC [12196] LOG: listening on IPv4 address "127.0.98.20", port 57423
2026-09-24 03:04:20.581 UTC [12196] LOG: listening on Unix socket "/tmp/.s.PGSQL.57423"
2026-09-24 03:04:20.607 UTC [12199] LOG: database system was shut down at 2026-09-24 03:04:18 UTC
2026-09-24 03:04:20.611 UTC [12196] LOG: database system is ready to accept connections
I20260924 03:04:21.587913 392 mini_postgres.cc:96] Postgres bound to 57423
2026-09-24 03:04:21.595 UTC [12207] FATAL: database "slave" does not exist
127.0.98.20:57423 - accepting connections
I20260924 03:04:21.596335 392 mini_ranger.cc:161] Starting Ranger...
I20260924 03:04:21.613823 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 03:04:22.269603 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 03:04:22.269709 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ranger-admin
2026-09-24 03:04:22,568 [I] DB FLAVOR :POSTGRES
2026-09-24 03:04:22,569 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 03:04:22,569 [I] Checking connection..
2026-09-24 03:04:22,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 03:04:22,824 [I] Checking connection passed.
2026-09-24 03:04:22,824 [I] --------- Verifying version history table ---------
2026-09-24 03:04:22,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:04:23,095 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 03:04:23,096 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:04:23,438 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 03:04:23,438 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 03:04:23,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57423/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 03:04:23.718 UTC [12271] WARNING: there is no transaction in progress
2026-09-24 03:04:23,726 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 03:04:23,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:04:24,002 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 03:04:24,002 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 03:04:24,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 03:04:24,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 03:04:24,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wgsb.c.gcp-upstream.internal','N') ;"
2026-09-24 03:04:24,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 03:04:25,170 [I] Table x_portal_user does not exist in database ranger
2026-09-24 03:04:25,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 03:04:25,435 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 03:04:25,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 03:04:25,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 03:04:25,984 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 03:04:25,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57423/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 03:04:27.112 UTC [12379] WARNING: there is no transaction in progress
2026-09-24 03:04:27.132 UTC [12379] WARNING: there is no transaction in progress
2026-09-24 03:04:27.152 UTC [12379] WARNING: there is no transaction in progress
2026-09-24 03:04:27.170 UTC [12379] WARNING: there is no transaction in progress
2026-09-24 03:04:27.191 UTC [12379] WARNING: there is no transaction in progress
2026-09-24 03:04:27.227 UTC [12379] WARNING: there is no transaction in progress
2026-09-24 03:04:27.297 UTC [12379] WARNING: there is no transaction in progress
2026-09-24 03:04:27.306 UTC [12379] WARNING: there is no transaction in progress
2026-09-24 03:04:27.324 UTC [12379] WARNING: there is no transaction in progress
2026-09-24 03:04:27.336 UTC [12379] WARNING: there is no transaction in progress
2026-09-24 03:04:27.347 UTC [12379] WARNING: there is no transaction in progress
2026-09-24 03:04:27.364 UTC [12379] WARNING: there is no transaction in progress
2026-09-24 03:04:27.375 UTC [12379] WARNING: there is no transaction in progress
2026-09-24 03:04:27.384 UTC [12379] WARNING: there is no transaction in progress
2026-09-24 03:04:27.394 UTC [12379] WARNING: there is no transaction in progress
2026-09-24 03:04:27.403 UTC [12379] WARNING: there is no transaction in progress
2026-09-24 03:04:28,254 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 03:04:28,254 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 03:04:28,532 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 03:04:28,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 03:04:28,815 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 03:04:28,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 03:04:29,071 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 03:04:29,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 03:04:29,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 03:04:29,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wgsb.c.gcp-upstream.internal';"
2026-09-24 03:04:29,916 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 03:04:29,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 03:04:30,196 [I] DB_PATCHES have already been applied
I20260924 03:04:30.202530 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskptPCbv/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 03:04:30.202612 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 03:04:30.205822 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:53257
Sep 24, 2026 3:04:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:04:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:04:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 3:04:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 3:04:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 3:04:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 3:04:31 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 03:04:31.324393 392 mini_ranger.cc:160] Time spent starting Ranger: real 9.728s user 0.000s sys 0.009s
Sep 24, 2026 3:04:54 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 3:04:54 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 3:04:54 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 3:04:54 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 3:04:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 3:04:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 3:04:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 3:04:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 3:04:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 3:04:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 3:04:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 3:04:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 3:04:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 3:04:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 3:04:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 3:04:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 3:04:54 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 3:04:55 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 03:04:55.720108 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 03:04:55 dist-test-slave-wgsb krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219095, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 03:04:55.807401 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 03:04:55.816654 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 03:04:56.970343 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 03:04:56.970427 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ranger-kms
I20260924 03:04:56.970441 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:57423
2026-09-24 03:04:57,160 [I] DB FLAVOR :POSTGRES
2026-09-24 03:04:57,160 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 03:04:57,160 [I] Checking connection
2026-09-24 03:04:57,160 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57423/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 03:04:57,430 [I] connection success
2026-09-24 03:04:57,430 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 03:04:57,430 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 03:04:57,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57423/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 03:04:57,701 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 03:04:57,701 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 03:04:57,702 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 03:04:57,702 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57423/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 03:04:58,036 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 03:04:58.679473 392 mini_ranger_kms.cc:310] Created kms service
I20260924 03:04:58.884016 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 03:04:58.938246 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 03:04:58 dist-test-slave-wgsb krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219095, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:04:59.432973 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 03:04:59.433180 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskptPCbv/build/release/bin/postgresql.jar:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskptPCbv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 03:04:59.433202 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 03:04:59.437537 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12560
I20260924 03:04:59.437628 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:35229
Sep 24, 2026 3:05:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:05:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:05:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 3:05:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 3:05:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 3:05:01 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 03:05:01.753541 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.946s user 0.002s sys 0.166s
I20260924 03:05:01.753620 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 03:05:01.753656 392 mini_ranger_kms.cc:401] 127.0.98.20:35229/kms/v1/keys
Sep 24 03:05:01 dist-test-slave-wgsb krb5kdc[12166](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790219101, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 03:05:02 dist-test-slave-wgsb krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219101, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 03:05:02 dist-test-slave-wgsb krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219101, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 03:05:05 dist-test-slave-wgsb krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:05:05.171235 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39795
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39795
--ranger_config_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35229/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:05:05.258328 12610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:05.258555 12610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:05.258579 12610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:05.259894 12610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:05:05.259943 12610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:05.259964 12610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:05.259977 12610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:05:05.259989 12610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:05:05.261736 12610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:35229/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39795
--ranger_config_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:05.262018 12610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:05.262295 12610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:05.273085 12618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:05.273208 12616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:05.273298 12615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:05.273536 12610 server_base.cc:1034] running on GCE node
I20260924 03:05:05.273867 12610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:05.274359 12610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:05.275563 12610 hybrid_clock.cc:648] HybridClock initialized: now 1790219105275538 us; error 37 us; skew 500 ppm
Sep 24 03:05:05 dist-test-slave-wgsb krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:05.284802 12610 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:05:05.285311 12610 webserver.cc:460] Webserver started at http://127.0.98.62:36829/ using document root <none> and password file <none>
I20260924 03:05:05.285540 12610 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:05.285601 12610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:05.285775 12610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:05:05 dist-test-slave-wgsb krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:05:05.340873 12610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "c90618d753114d59a04c2e7ad08475e3"
format_stamp: "Formatted at 2026-09-24 03:05:05 on dist-test-slave-wgsb"
server_key: "b314803fc192480d9f1f5ac7a73ba706"
server_key_iv: "2c3e76c16e98ef15de5e94b653f507ff"
server_key_version: "kuduclusterkey@0"
I20260924 03:05:05.341351 12610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "c90618d753114d59a04c2e7ad08475e3"
format_stamp: "Formatted at 2026-09-24 03:05:05 on dist-test-slave-wgsb"
server_key: "b314803fc192480d9f1f5ac7a73ba706"
server_key_iv: "2c3e76c16e98ef15de5e94b653f507ff"
server_key_version: "kuduclusterkey@0"
I20260924 03:05:05.343134 12610 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260924 03:05:05.358115 12625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:05.358342 12610 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:05:05.358438 12610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "c90618d753114d59a04c2e7ad08475e3"
format_stamp: "Formatted at 2026-09-24 03:05:05 on dist-test-slave-wgsb"
server_key: "b314803fc192480d9f1f5ac7a73ba706"
server_key_iv: "2c3e76c16e98ef15de5e94b653f507ff"
server_key_version: "kuduclusterkey@0"
I20260924 03:05:05.358544 12610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:05.384790 12610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:05.387041 12610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:05.387202 12610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:05.392719 12610 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39795
I20260924 03:05:05.392745 12677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39795 every 8 connection(s)
I20260924 03:05:05.393173 12610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:05:05.393877 12678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:05:05.396334 12678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3: Bootstrap starting.
I20260924 03:05:05.396922 12678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3: Neither blocks nor log segments found. Creating new log.
I20260924 03:05:05.397217 12678 log.cc:824] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3: Log is configured to *not* fsync() on all Append() calls
I20260924 03:05:05.397919 12678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3: No bootstrap required, opened a new log
I20260924 03:05:05.400048 12678 raft_consensus.cc:348] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c90618d753114d59a04c2e7ad08475e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39795 } }
I20260924 03:05:05.400223 12678 raft_consensus.cc:374] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:05:05.400269 12678 raft_consensus.cc:729] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c90618d753114d59a04c2e7ad08475e3, State: Initialized, Role: FOLLOWER
I20260924 03:05:05.400454 12678 consensus_queue.cc:260] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c90618d753114d59a04c2e7ad08475e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39795 } }
I20260924 03:05:05.400552 12678 raft_consensus.cc:388] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:05:05.400596 12678 raft_consensus.cc:482] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:05:05.400642 12678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:05:05.401167 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 12610
I20260924 03:05:05.401255 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:05:05.401211 12678 raft_consensus.cc:504] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c90618d753114d59a04c2e7ad08475e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39795 } }
I20260924 03:05:05.401358 12678 leader_election.cc:302] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c90618d753114d59a04c2e7ad08475e3; no voters:
I20260924 03:05:05.401425 392 external_mini_cluster.cc:1428] Setting key 993eaa15ebb86227b53570ed8d118d2c
I20260924 03:05:05.401553 12678 leader_election.cc:288] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:05:05.401582 12681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:05:05.401979 12681 raft_consensus.cc:686] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [term 1 LEADER]: Becoming Leader. State: Replica: c90618d753114d59a04c2e7ad08475e3, State: Running, Role: LEADER
I20260924 03:05:05.401829 12678 sys_catalog.cc:564] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:05:05.402163 12681 consensus_queue.cc:237] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c90618d753114d59a04c2e7ad08475e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39795 } }
I20260924 03:05:05.402575 12681 sys_catalog.cc:455] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c90618d753114d59a04c2e7ad08475e3. Latest consensus state: current_term: 1 leader_uuid: "c90618d753114d59a04c2e7ad08475e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c90618d753114d59a04c2e7ad08475e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39795 } } }
I20260924 03:05:05.402711 12681 sys_catalog.cc:458] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:05:05.403652 12684 sys_catalog.cc:455] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c90618d753114d59a04c2e7ad08475e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c90618d753114d59a04c2e7ad08475e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39795 } } }
I20260924 03:05:05.403782 12684 sys_catalog.cc:458] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:05:05.403977 12687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:05:05.404305 12678 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 03:05:05.404480 12687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 03:05:05 dist-test-slave-wgsb krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:05:05.412752 12687 catalog_manager.cc:1347] Generated new cluster ID: f4fed588aa8249ebb292cea14f484877
I20260924 03:05:05.412853 12687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:05:05.417927 12685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:05.401995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48190 (local address 127.0.98.62:39795)
0924 03:05:05.402180 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:05.402191 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:05.402224 (+ 33us) server_negotiation.cc:407] Connection header received
0924 03:05:05.402345 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:05.402352 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:05.402408 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:05.402545 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:05.403393 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:05.406866 (+ 3473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:05.408105 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:05.409268 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:05.409419 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:05.413411 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:05.413416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:05.413416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:05.413430 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:05.415567 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:05.416060 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:05.416063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:05.416064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:05.416203 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:05.416460 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:05.416461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:05.416461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:05.416828 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:05.416988 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:05.417533 (+ 545us) server_negotiation.cc:299] Negotiation successful
0924 03:05:05.417672 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
I20260924 03:05:05.433089 12687 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:05:05.433918 12687 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:05:05.441323 12687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3: Generated new TSK 0
I20260924 03:05:05.441625 12687 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:05:06.225582 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39795
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:05:06.341851 12723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:06.342115 12723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:06.342160 12723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:06.343389 12723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:06.343461 12723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:06.343508 12723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:05:06.345232 12723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:35229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:06.345525 12723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:06.346616 12723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:06.352029 12729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:06.352095 12731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:06.352284 12723 server_base.cc:1034] running on GCE node
W20260924 03:05:06.352537 12728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:06.352773 12723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:06.353053 12723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:06.354259 12723 hybrid_clock.cc:648] HybridClock initialized: now 1790219106354239 us; error 36 us; skew 500 ppm
Sep 24 03:05:06 dist-test-slave-wgsb krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:06.357153 12723 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:05:06.357635 12723 webserver.cc:460] Webserver started at http://127.0.98.1:36433/ using document root <none> and password file <none>
I20260924 03:05:06.357932 12723 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:06.358004 12723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:06.358150 12723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:05:06 dist-test-slave-wgsb krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:05:06.371939 12723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/data/instance:
uuid: "fd6fc90bdddf4ef5b4c2067105ae9287"
format_stamp: "Formatted at 2026-09-24 03:05:06 on dist-test-slave-wgsb"
server_key: "f16a40a2080394ce456ea88b6bf52749"
server_key_iv: "001f037cee80682c1017456bbeb29bdb"
server_key_version: "kuduclusterkey@0"
I20260924 03:05:06.372444 12723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fd6fc90bdddf4ef5b4c2067105ae9287"
format_stamp: "Formatted at 2026-09-24 03:05:06 on dist-test-slave-wgsb"
server_key: "f16a40a2080394ce456ea88b6bf52749"
server_key_iv: "001f037cee80682c1017456bbeb29bdb"
server_key_version: "kuduclusterkey@0"
I20260924 03:05:06.384145 12723 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.001s sys 0.000s
I20260924 03:05:06.402959 12740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:06.403231 12723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260924 03:05:06.403323 12723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "fd6fc90bdddf4ef5b4c2067105ae9287"
format_stamp: "Formatted at 2026-09-24 03:05:06 on dist-test-slave-wgsb"
server_key: "f16a40a2080394ce456ea88b6bf52749"
server_key_iv: "001f037cee80682c1017456bbeb29bdb"
server_key_version: "kuduclusterkey@0"
I20260924 03:05:06.403416 12723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:06.413869 12723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:06.416312 12723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:06.416503 12723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:06.416740 12723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:05:06.417145 12723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:05:06.417199 12723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:06.417258 12723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:05:06.417295 12723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:06.425757 12723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41019
I20260924 03:05:06.426211 12723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:05:06.426833 12853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41019 every 8 connection(s)
Sep 24 03:05:06 dist-test-slave-wgsb krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:05:06.434711 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 12723
I20260924 03:05:06.434819 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:05:06.435041 392 external_mini_cluster.cc:1428] Setting key db406a882229bee46f4482a141df0d63
I20260924 03:05:06.439000 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:06.427054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39795 (local address 127.0.98.1:54119)
0924 03:05:06.427698 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:06.427744 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:06.428264 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:06.428664 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:06.428674 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:06.428878 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:06.429094 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:06.429098 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:06.429874 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:06.429875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:06.431188 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:06.431191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:06.431847 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:06.431848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:06.432024 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:06.432681 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:06.432692 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:06.434283 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:06.437018 (+ 2735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:06.437026 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:06.437028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:06.437385 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:06.437634 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:06.437635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:06.437636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:06.437802 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:06.438152 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:06.438158 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:06.438361 (+ 203us) client_negotiation.cc:769] Sending connection context
0924 03:05:06.438483 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 03:05:06.438675 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":160,"threads_started":1}
I20260924 03:05:06.439710 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:06.427059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54119 (local address 127.0.98.62:39795)
0924 03:05:06.428418 (+ 1359us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:06.428420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:06.428427 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:05:06.428459 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:06.428460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:06.428471 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:06.428557 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:06.429207 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:06.429772 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:06.431301 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:06.431683 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:06.431756 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:06.434387 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:06.434389 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:06.434390 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:06.434398 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:06.436903 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:06.437482 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:06.437484 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:06.437484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:06.437555 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:06.437879 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:06.437880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:06.437881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:06.438051 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:06.438121 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:06.439415 (+ 1294us) server_negotiation.cc:299] Negotiation successful
0924 03:05:06.439521 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1330,"thread_start_us":101,"threads_started":1}
I20260924 03:05:06.440747 12854 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39795
I20260924 03:05:06.440853 12854 heartbeater.cc:464] Registering TS with master...
I20260924 03:05:06.441109 12854 heartbeater.cc:511] Master 127.0.98.62:39795 requested a full tablet report, sending...
I20260924 03:05:06.441779 12642 ts_manager.cc:194] Registered new tserver with Master: fd6fc90bdddf4ef5b4c2067105ae9287 (127.0.98.1:41019)
I20260924 03:05:06.442857 12642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54119
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:05:06.471072 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39795
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:05:06 dist-test-slave-wgsb krb5kdc[12166](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790219106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 03:05:06.598685 12860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:06.599081 12860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:06.599138 12860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:06.601099 12860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:06.601223 12860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:06.601312 12860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:05:06.603701 12860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:35229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:06.603971 12860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:06.604300 12860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:06.610786 12868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:06.610991 12870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:06.611274 12867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:06.611831 12860 server_base.cc:1034] running on GCE node
I20260924 03:05:06.612039 12860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:06.612308 12860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:06.613466 12860 hybrid_clock.cc:648] HybridClock initialized: now 1790219106613450 us; error 34 us; skew 500 ppm
Sep 24 03:05:06 dist-test-slave-wgsb krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:06.616786 12860 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:05:06.617376 12860 webserver.cc:460] Webserver started at http://127.0.98.2:38557/ using document root <none> and password file <none>
I20260924 03:05:06.617661 12860 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:06.617724 12860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:06.617859 12860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:05:06 dist-test-slave-wgsb krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:05:06.631714 12860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/data/instance:
uuid: "e038465b23f44d66ba0400ddfbcae0c3"
format_stamp: "Formatted at 2026-09-24 03:05:06 on dist-test-slave-wgsb"
server_key: "666cc617712f3238a77e548716fb3ae8"
server_key_iv: "8806f465095f7b4ed15bce1d53d3f556"
server_key_version: "kuduclusterkey@0"
I20260924 03:05:06.632145 12860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e038465b23f44d66ba0400ddfbcae0c3"
format_stamp: "Formatted at 2026-09-24 03:05:06 on dist-test-slave-wgsb"
server_key: "666cc617712f3238a77e548716fb3ae8"
server_key_iv: "8806f465095f7b4ed15bce1d53d3f556"
server_key_version: "kuduclusterkey@0"
I20260924 03:05:06.633728 12860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:05:06.651841 12878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:06.652112 12860 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:05:06.652220 12860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "e038465b23f44d66ba0400ddfbcae0c3"
format_stamp: "Formatted at 2026-09-24 03:05:06 on dist-test-slave-wgsb"
server_key: "666cc617712f3238a77e548716fb3ae8"
server_key_iv: "8806f465095f7b4ed15bce1d53d3f556"
server_key_version: "kuduclusterkey@0"
I20260924 03:05:06.652323 12860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:06.673453 12860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:06.676019 12860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:06.676255 12860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:06.676529 12860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:05:06.677139 12860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:05:06.677234 12860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:06.677335 12860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:05:06.677400 12860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:06.684262 12860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46727
I20260924 03:05:06.685096 12860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:05:06.691402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 12860
I20260924 03:05:06.691509 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:05:06.691746 392 external_mini_cluster.cc:1428] Setting key 4c46ec3d5b0518128d547ead3cd110c2
I20260924 03:05:06.693369 12991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46727 every 8 connection(s)
Sep 24 03:05:06 dist-test-slave-wgsb krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:05:06.706163 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:06.694364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51081 (local address 127.0.98.62:39795)
0924 03:05:06.694424 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:06.694426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:06.694695 (+ 269us) server_negotiation.cc:407] Connection header received
0924 03:05:06.695651 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:06.695653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:06.695668 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:06.695702 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:06.696666 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:06.697438 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:06.699072 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:06.699393 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:06.699441 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:06.702468 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:06.702470 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:06.702471 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:06.702480 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:06.704337 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:06.704885 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:06.704887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:06.704887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:06.704949 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:06.705279 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:06.705280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:06.705281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:06.705440 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:06.705517 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:06.705955 (+ 438us) server_negotiation.cc:299] Negotiation successful
0924 03:05:06.706059 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 03:05:06.707048 12995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:06.694056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39795 (local address 127.0.98.2:51081)
0924 03:05:06.694556 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:06.694606 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:06.695442 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:06.695963 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:06.695969 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:06.696225 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:06.696525 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:06.696532 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:06.697573 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:06.697574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:06.698964 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:06.698967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:06.699542 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:06.699543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:06.699747 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:06.700545 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:06.700567 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:06.702333 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:06.704442 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:06.704451 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:06.704453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:06.704786 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:06.705024 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:06.705025 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:06.705025 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:06.705185 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:06.705548 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:06.705555 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:06.705784 (+ 229us) client_negotiation.cc:769] Sending connection context
0924 03:05:06.706569 (+ 785us) client_negotiation.cc:240] Negotiation successful
0924 03:05:06.706793 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":103,"threads_started":1}
I20260924 03:05:06.707806 12992 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39795
I20260924 03:05:06.707907 12992 heartbeater.cc:464] Registering TS with master...
I20260924 03:05:06.708104 12992 heartbeater.cc:511] Master 127.0.98.62:39795 requested a full tablet report, sending...
I20260924 03:05:06.708489 12642 ts_manager.cc:194] Registered new tserver with Master: e038465b23f44d66ba0400ddfbcae0c3 (127.0.98.2:46727)
I20260924 03:05:06.709012 12642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51081
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:05:06.735900 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39795
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:05:06.875213 12997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:06.875535 12997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:06.875574 12997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:06.877502 12997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:06.877575 12997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:06.877617 12997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:05:06.879922 12997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:35229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:06.880216 12997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:06.880506 12997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:06.887063 13002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:06.887072 13005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:06.887519 13003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:06.888162 12997 server_base.cc:1034] running on GCE node
I20260924 03:05:06.888386 12997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:06.888660 12997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:06.889823 12997 hybrid_clock.cc:648] HybridClock initialized: now 1790219106889796 us; error 46 us; skew 500 ppm
Sep 24 03:05:06 dist-test-slave-wgsb krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:06.893247 12997 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:05:06.893903 12997 webserver.cc:460] Webserver started at http://127.0.98.3:36979/ using document root <none> and password file <none>
I20260924 03:05:06.894214 12997 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:06.894290 12997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:06.894460 12997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:05:06 dist-test-slave-wgsb krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:05:06.917632 12997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/data/instance:
uuid: "985ebedc2f6c4f9389abd4c27309de48"
format_stamp: "Formatted at 2026-09-24 03:05:06 on dist-test-slave-wgsb"
server_key: "e058ea48c1c77b2bd2e2eda8670e568f"
server_key_iv: "53db8d62b95bf91b6fe22c528887e0ee"
server_key_version: "kuduclusterkey@0"
I20260924 03:05:06.918111 12997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/wal/instance:
uuid: "985ebedc2f6c4f9389abd4c27309de48"
format_stamp: "Formatted at 2026-09-24 03:05:06 on dist-test-slave-wgsb"
server_key: "e058ea48c1c77b2bd2e2eda8670e568f"
server_key_iv: "53db8d62b95bf91b6fe22c528887e0ee"
server_key_version: "kuduclusterkey@0"
I20260924 03:05:06.924846 12997 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.003s
I20260924 03:05:06.949513 13012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:06.949759 12997 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.000s sys 0.000s
I20260924 03:05:06.949863 12997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "985ebedc2f6c4f9389abd4c27309de48"
format_stamp: "Formatted at 2026-09-24 03:05:06 on dist-test-slave-wgsb"
server_key: "e058ea48c1c77b2bd2e2eda8670e568f"
server_key_iv: "53db8d62b95bf91b6fe22c528887e0ee"
server_key_version: "kuduclusterkey@0"
I20260924 03:05:06.949978 12997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:06.962592 12997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:06.965281 12997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:06.965442 12997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:06.965693 12997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:05:06.966079 12997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:05:06.966130 12997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:06.966168 12997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:05:06.966192 12997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:06.974174 12997 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41107
I20260924 03:05:06.974649 12997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:05:06.977895 13125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41107 every 8 connection(s)
I20260924 03:05:06.978744 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 12997
I20260924 03:05:06.978837 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:05:06.979033 392 external_mini_cluster.cc:1428] Setting key ca72c062ebed5101f8c8c7824d247ca5
Sep 24 03:05:06 dist-test-slave-wgsb krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:05:06.989562 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:06.977404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48081 (local address 127.0.98.62:39795)
0924 03:05:06.977450 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:06.977453 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:06.978259 (+ 806us) server_negotiation.cc:407] Connection header received
0924 03:05:06.979021 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:06.979022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:06.979036 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:06.979069 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:06.979983 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:06.980756 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:06.982288 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:06.982580 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:06.982629 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:06.985821 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:06.985823 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:06.985824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:06.985834 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:06.987738 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:06.988329 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:06.988330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:06.988331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:06.988397 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:06.988730 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:06.988731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:06.988732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:06.988888 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:06.988935 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:06.989363 (+ 428us) server_negotiation.cc:299] Negotiation successful
0924 03:05:06.989456 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:05:06.990166 13128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:06.977095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39795 (local address 127.0.98.3:48081)
0924 03:05:06.978113 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:06.978158 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:06.978880 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:06.979276 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:06.979281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:06.979540 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:06.979867 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:06.979874 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:06.980877 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:06.980878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:06.982182 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:06.982185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:06.982739 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:06.982740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:06.982886 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:06.983546 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:06.983568 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:06.985664 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:06.987856 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:06.987867 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:06.987869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:06.988221 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:06.988480 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:06.988481 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:06.988481 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:06.988646 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:06.988986 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:06.988994 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:06.989193 (+ 199us) client_negotiation.cc:769] Sending connection context
0924 03:05:06.989737 (+ 544us) client_negotiation.cc:240] Negotiation successful
0924 03:05:06.989919 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974,"thread_start_us":106,"threads_started":1}
I20260924 03:05:06.990593 13126 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39795
I20260924 03:05:06.990701 13126 heartbeater.cc:464] Registering TS with master...
I20260924 03:05:06.990900 13126 heartbeater.cc:511] Master 127.0.98.62:39795 requested a full tablet report, sending...
I20260924 03:05:06.991238 12642 ts_manager.cc:194] Registered new tserver with Master: 985ebedc2f6c4f9389abd4c27309de48 (127.0.98.3:41107)
I20260924 03:05:06.991765 12642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48081
I20260924 03:05:07.001919 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:05:07.002866 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 12723
I20260924 03:05:07.010113 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 12860
I20260924 03:05:07.017508 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 12997
I20260924 03:05:07.024490 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 12610
I20260924 03:05:07.031495 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39795
--webserver_interface=127.0.98.62
--webserver_port=36829
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39795
--ranger_config_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35229/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:35229/kms with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:05:07.115412 13129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:07.115676 13129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:07.115713 13129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:07.116953 13129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:05:07.116999 13129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:07.117014 13129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:07.117026 13129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:05:07.117038 13129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:05:07.118637 13129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:35229/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39795
--ranger_config_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:07.118882 13129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:07.119133 13129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:07.124815 13137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:07.124815 13135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:07.124883 13134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:07.125165 13129 server_base.cc:1034] running on GCE node
I20260924 03:05:07.125535 13129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:07.125870 13129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:07.127162 13129 hybrid_clock.cc:648] HybridClock initialized: now 1790219107127142 us; error 37 us; skew 500 ppm
Sep 24 03:05:07 dist-test-slave-wgsb krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:07.130167 13129 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:05:07.130764 13129 webserver.cc:460] Webserver started at http://127.0.98.62:36829/ using document root <none> and password file <none>
I20260924 03:05:07.131023 13129 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:07.131080 13129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 03:05:07 dist-test-slave-wgsb krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:05:07.179894 13129 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:05:07.180848 13145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:07.181144 13129 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:05:07.181253 13129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "c90618d753114d59a04c2e7ad08475e3"
format_stamp: "Formatted at 2026-09-24 03:05:05 on dist-test-slave-wgsb"
server_key: "b314803fc192480d9f1f5ac7a73ba706"
server_key_iv: "2c3e76c16e98ef15de5e94b653f507ff"
server_key_version: "kuduclusterkey@0"
I20260924 03:05:07.181751 13129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:07.205080 13129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:07.206904 13129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:07.207086 13129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:07.211776 13129 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39795
I20260924 03:05:07.211812 13197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39795 every 8 connection(s)
I20260924 03:05:07.212235 13129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:05:07.213441 13198 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:05:07.214138 13198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3: Bootstrap starting.
I20260924 03:05:07.217756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 13129
I20260924 03:05:07.218091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41019
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39795
--webserver_port=36433
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:35229/kms with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:05:07.222287 13198 log.cc:824] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3: Log is configured to *not* fsync() on all Append() calls
I20260924 03:05:07.224143 13198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:05:07.224432 13198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3: Bootstrap complete.
I20260924 03:05:07.226868 13198 raft_consensus.cc:348] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c90618d753114d59a04c2e7ad08475e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39795 } }
I20260924 03:05:07.227154 13198 raft_consensus.cc:729] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c90618d753114d59a04c2e7ad08475e3, State: Initialized, Role: FOLLOWER
I20260924 03:05:07.227355 13198 consensus_queue.cc:260] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c90618d753114d59a04c2e7ad08475e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39795 } }
I20260924 03:05:07.227448 13198 raft_consensus.cc:388] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:05:07.227499 13198 raft_consensus.cc:482] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:05:07.227547 13198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:05:07.228595 13198 raft_consensus.cc:504] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c90618d753114d59a04c2e7ad08475e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39795 } }
I20260924 03:05:07.228714 13198 leader_election.cc:302] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c90618d753114d59a04c2e7ad08475e3; no voters:
I20260924 03:05:07.228883 13198 leader_election.cc:288] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 03:05:07.228965 13202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 03:05:07.229146 13198 sys_catalog.cc:564] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:05:07.229190 13202 raft_consensus.cc:686] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [term 2 LEADER]: Becoming Leader. State: Replica: c90618d753114d59a04c2e7ad08475e3, State: Running, Role: LEADER
I20260924 03:05:07.229386 13202 consensus_queue.cc:237] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c90618d753114d59a04c2e7ad08475e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39795 } }
I20260924 03:05:07.229700 13203 sys_catalog.cc:455] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c90618d753114d59a04c2e7ad08475e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c90618d753114d59a04c2e7ad08475e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39795 } } }
I20260924 03:05:07.229768 13203 sys_catalog.cc:458] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:05:07.230033 13204 sys_catalog.cc:455] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c90618d753114d59a04c2e7ad08475e3. Latest consensus state: current_term: 2 leader_uuid: "c90618d753114d59a04c2e7ad08475e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c90618d753114d59a04c2e7ad08475e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39795 } } }
I20260924 03:05:07.230085 13204 sys_catalog.cc:458] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:05:07.230397 13210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:05:07.230737 13198 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 03:05:07.230938 13210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:05:07.231073 13210 catalog_manager.cc:1259] Loaded cluster ID: f4fed588aa8249ebb292cea14f484877
I20260924 03:05:07.231114 13210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:05:07.235638 13210 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:05:07.236146 13210 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3: Loaded TSK: 0
I20260924 03:05:07.236445 13210 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 03:05:07.313462 13200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:07.313762 13200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:07.313805 13200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:07.315886 13200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:07.316105 13200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:07.316195 13200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:05:07.319166 13200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:35229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41019
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36433
--tserver_master_addrs=127.0.98.62:39795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:07.319656 13200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:07.319944 13200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:07.326763 13239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:07.326758 13236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:07.326774 13237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:07.327103 13200 server_base.cc:1034] running on GCE node
I20260924 03:05:07.327400 13200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:07.327667 13200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:07.328814 13200 hybrid_clock.cc:648] HybridClock initialized: now 1790219107328792 us; error 35 us; skew 500 ppm
Sep 24 03:05:07 dist-test-slave-wgsb krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:07.332171 13200 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:05:07.332707 13200 webserver.cc:460] Webserver started at http://127.0.98.1:36433/ using document root <none> and password file <none>
I20260924 03:05:07.332971 13200 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:07.333088 13200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 03:05:07 dist-test-slave-wgsb krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:05:07.361634 13200 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 03:05:07.362454 13247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:07.362654 13200 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 03:05:07.362737 13200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "fd6fc90bdddf4ef5b4c2067105ae9287"
format_stamp: "Formatted at 2026-09-24 03:05:06 on dist-test-slave-wgsb"
server_key: "f16a40a2080394ce456ea88b6bf52749"
server_key_iv: "001f037cee80682c1017456bbeb29bdb"
server_key_version: "kuduclusterkey@0"
I20260924 03:05:07.363161 13200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:07.374783 13200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:07.377270 13200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:07.377473 13200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:07.377770 13200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:05:07.378171 13200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:05:07.378271 13200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:07.378368 13200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:05:07.378434 13200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:07.385099 13200 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41019
I20260924 03:05:07.385622 13200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:05:07.386482 13360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41019 every 8 connection(s)
I20260924 03:05:07.393944 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 13200
I20260924 03:05:07.394250 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46727
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39795
--webserver_port=38557
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:35229/kms with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:05:07 dist-test-slave-wgsb krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:05:07.402426 13363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:07.386428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39795 (local address 127.0.98.1:41041)
0924 03:05:07.387704 (+ 1276us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:07.387754 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:07.388479 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:07.389188 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:07.389192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:07.389433 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:07.389687 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:07.389692 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:07.391383 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:07.391383 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:07.392694 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:07.392696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:07.393710 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:07.393711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:07.393902 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:07.394544 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:07.394554 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:07.398108 (+ 3554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:07.400310 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:07.400318 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:07.400321 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:07.400666 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:07.401002 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:07.401003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:07.401003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:07.401172 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:07.401697 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:07.401704 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:07.401887 (+ 183us) client_negotiation.cc:769] Sending connection context
0924 03:05:07.401988 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 03:05:07.402177 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1230,"thread_start_us":83,"threads_started":1}
I20260924 03:05:07.403319 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:07.386520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41041 (local address 127.0.98.62:39795)
0924 03:05:07.386800 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:07.386811 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:07.387866 (+ 1055us) server_negotiation.cc:407] Connection header received
0924 03:05:07.388754 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:07.388763 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:07.388825 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:07.388945 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:07.389813 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:07.391254 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:07.392796 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:07.393491 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:07.393573 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:07.398286 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:07.398291 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:07.398292 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:07.398307 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:07.400172 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:07.400780 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:07.400784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:07.400785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:07.400914 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:07.401251 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:07.401251 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:07.401252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:07.401571 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:07.401665 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:07.402920 (+ 1255us) server_negotiation.cc:299] Negotiation successful
0924 03:05:07.403058 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":82,"threads_started":1}
I20260924 03:05:07.404567 13361 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39795
I20260924 03:05:07.404663 13361 heartbeater.cc:464] Registering TS with master...
I20260924 03:05:07.404852 13361 heartbeater.cc:511] Master 127.0.98.62:39795 requested a full tablet report, sending...
W20260924 03:05:07.405812 13361 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39795 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 03:05:07.405880 13361 heartbeater.cc:464] Registering TS with master...
I20260924 03:05:07.405943 13361 heartbeater.cc:511] Master 127.0.98.62:39795 requested a full tablet report, sending...
I20260924 03:05:07.406358 13361 heartbeater.cc:464] Registering TS with master...
I20260924 03:05:07.406433 13361 heartbeater.cc:511] Master 127.0.98.62:39795 requested a full tablet report, sending...
W20260924 03:05:07.406792 13361 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:05:07.509819 13365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:07.510057 13365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:07.510099 13365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:07.511530 13365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:07.511596 13365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:07.511665 13365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:05:07.513383 13365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:35229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38557
--tserver_master_addrs=127.0.98.62:39795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:07.513638 13365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:07.513859 13365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:07.519369 13373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:07.519696 13365 server_base.cc:1034] running on GCE node
W20260924 03:05:07.519755 13370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:07.519876 13371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:07.520179 13365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:07.520445 13365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:07.523675 13365 hybrid_clock.cc:648] HybridClock initialized: now 1790219107523640 us; error 52 us; skew 500 ppm
Sep 24 03:05:07 dist-test-slave-wgsb krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:07.526305 13365 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:05:07.526885 13365 webserver.cc:460] Webserver started at http://127.0.98.2:38557/ using document root <none> and password file <none>
I20260924 03:05:07.527158 13365 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:07.527235 13365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 03:05:07 dist-test-slave-wgsb krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:05:07.544054 13365 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:05:07.545006 13381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:07.545225 13365 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:05:07.545306 13365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "e038465b23f44d66ba0400ddfbcae0c3"
format_stamp: "Formatted at 2026-09-24 03:05:06 on dist-test-slave-wgsb"
server_key: "666cc617712f3238a77e548716fb3ae8"
server_key_iv: "8806f465095f7b4ed15bce1d53d3f556"
server_key_version: "kuduclusterkey@0"
I20260924 03:05:07.545782 13365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:07.570715 13365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:07.573159 13365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:07.573320 13365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:07.573598 13365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:05:07.573980 13365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:05:07.574034 13365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:07.574079 13365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:05:07.574147 13365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:07.582155 13365 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46727
I20260924 03:05:07.582334 13494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46727 every 8 connection(s)
I20260924 03:05:07.582636 13365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 03:05:07 dist-test-slave-wgsb krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:05:07.591888 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 13365
I20260924 03:05:07.592214 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41107
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39795
--webserver_port=36979
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:35229/kms with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:05:07.596836 13497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:07.583477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39795 (local address 127.0.98.2:39091)
0924 03:05:07.583688 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:07.583743 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:07.584469 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:07.584774 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:07.584779 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:07.585074 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:07.585388 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:07.585395 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:07.586221 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:07.586222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:07.587260 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:07.587263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:07.587856 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:07.587858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:07.588012 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:07.588721 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:07.588731 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:07.590525 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:07.594801 (+ 4276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:07.594810 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:07.594813 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:07.595195 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:07.595464 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:07.595465 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:07.595465 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:07.595654 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:07.595988 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:07.595995 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:07.596214 (+ 219us) client_negotiation.cc:769] Sending connection context
0924 03:05:07.596342 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 03:05:07.596531 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":69,"threads_started":1}
I20260924 03:05:07.597347 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:07.583495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39091 (local address 127.0.98.62:39795)
0924 03:05:07.583557 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:07.583560 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:07.583840 (+ 280us) server_negotiation.cc:407] Connection header received
0924 03:05:07.584603 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:07.584605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:07.584619 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:07.584693 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:07.585488 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:07.586066 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:07.587359 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:07.587726 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:07.587810 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:07.590658 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:07.590661 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:07.590661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:07.590671 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:07.594680 (+ 4009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:07.595299 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:07.595300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:07.595301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:07.595376 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:07.595736 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:07.595737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:07.595738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:07.595900 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:07.595987 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:07.597123 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0924 03:05:07.597214 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 03:05:07.597818 13495 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39795
I20260924 03:05:07.597941 13495 heartbeater.cc:464] Registering TS with master...
I20260924 03:05:07.598170 13495 heartbeater.cc:511] Master 127.0.98.62:39795 requested a full tablet report, sending...
W20260924 03:05:07.598932 13495 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39795 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 03:05:07.599006 13495 heartbeater.cc:464] Registering TS with master...
I20260924 03:05:07.599081 13495 heartbeater.cc:511] Master 127.0.98.62:39795 requested a full tablet report, sending...
I20260924 03:05:07.599454 13495 heartbeater.cc:464] Registering TS with master...
I20260924 03:05:07.599526 13495 heartbeater.cc:511] Master 127.0.98.62:39795 requested a full tablet report, sending...
W20260924 03:05:07.599916 13495 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:05:07.700893 13498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:07.701253 13498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:07.701292 13498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:07.703142 13498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:07.703253 13498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:07.703347 13498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:05:07.705616 13498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:35229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36979
--tserver_master_addrs=127.0.98.62:39795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:07.705873 13498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:07.706175 13498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:07.712503 13506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:07.712508 13503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:07.713107 13504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:07.713236 13498 server_base.cc:1034] running on GCE node
I20260924 03:05:07.713574 13498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:07.713841 13498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:07.717073 13498 hybrid_clock.cc:648] HybridClock initialized: now 1790219107717039 us; error 44 us; skew 500 ppm
Sep 24 03:05:07 dist-test-slave-wgsb krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:07.720343 13498 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:05:07.720940 13498 webserver.cc:460] Webserver started at http://127.0.98.3:36979/ using document root <none> and password file <none>
I20260924 03:05:07.721282 13498 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:07.721374 13498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 03:05:07 dist-test-slave-wgsb krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:05:07.752449 13498 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260924 03:05:07.753443 13515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:07.753634 13498 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 03:05:07.753719 13498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "985ebedc2f6c4f9389abd4c27309de48"
format_stamp: "Formatted at 2026-09-24 03:05:06 on dist-test-slave-wgsb"
server_key: "e058ea48c1c77b2bd2e2eda8670e568f"
server_key_iv: "53db8d62b95bf91b6fe22c528887e0ee"
server_key_version: "kuduclusterkey@0"
I20260924 03:05:07.754165 13498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:07.769759 13498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:07.772275 13498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:07.772449 13498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:07.772668 13498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:05:07.773082 13498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:05:07.773135 13498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:07.773173 13498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:05:07.773202 13498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:07.782598 13498 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41107
I20260924 03:05:07.784408 13498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:05:07.785491 13628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41107 every 8 connection(s)
I20260924 03:05:07.786522 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 13498
Sep 24 03:05:07 dist-test-slave-wgsb krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:05:07.797000 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:07.785689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41807 (local address 127.0.98.62:39795)
0924 03:05:07.786259 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:07.786261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:07.786295 (+ 34us) server_negotiation.cc:407] Connection header received
0924 03:05:07.786985 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:07.786987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:07.787001 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:07.787035 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:07.788384 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:07.789187 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:07.790319 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:07.790534 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:07.790622 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:07.793349 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:07.793351 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:07.793351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:07.793359 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:07.795121 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:07.795669 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:07.795670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:07.795671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:07.795726 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:07.796041 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:07.796042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:07.796043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:07.796233 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:07.796315 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:07.796792 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 03:05:07.796883 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539}
I20260924 03:05:07.797534 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:07.785700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39795 (local address 127.0.98.3:41807)
0924 03:05:07.786163 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:07.786209 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:07.786852 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:07.787808 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:07.787812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:07.788022 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:07.788224 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:07.788227 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:07.789304 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:07.789305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:07.790223 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:07.790225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:07.790643 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:07.790644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:07.790784 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:07.791474 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:07.791485 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:07.793232 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:07.795218 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:07.795225 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:07.795227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:07.795552 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:07.795799 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:07.795801 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:07.795801 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:07.795946 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:07.796321 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:07.796327 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:07.796579 (+ 252us) client_negotiation.cc:769] Sending connection context
0924 03:05:07.797113 (+ 534us) client_negotiation.cc:240] Negotiation successful
0924 03:05:07.797283 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":383,"threads_started":1}
I20260924 03:05:07.797596 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:07.786974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48192 (local address 127.0.98.62:39795)
0924 03:05:07.788440 (+ 1466us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:07.788442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:07.788657 (+ 215us) server_negotiation.cc:407] Connection header received
0924 03:05:07.788686 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:07.788687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:07.788700 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:07.788773 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:07.789502 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:07.790238 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:07.791634 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:07.791958 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:07.792039 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:07.793472 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:07.793474 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:07.793474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:07.793482 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:07.795778 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:07.796391 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:07.796392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:07.796392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:07.796461 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:07.796706 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:07.796707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:07.796707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:07.796831 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:07.797012 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:07.797400 (+ 388us) server_negotiation.cc:299] Negotiation successful
0924 03:05:07.797491 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1444,"thread_start_us":69,"threads_started":1}
I20260924 03:05:07.797958 13629 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39795
I20260924 03:05:07.798053 13629 heartbeater.cc:464] Registering TS with master...
I20260924 03:05:07.798233 13629 heartbeater.cc:511] Master 127.0.98.62:39795 requested a full tablet report, sending...
W20260924 03:05:07.799003 13629 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39795 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 03:05:07.799088 13629 heartbeater.cc:464] Registering TS with master...
I20260924 03:05:07.799160 13629 heartbeater.cc:511] Master 127.0.98.62:39795 requested a full tablet report, sending...
I20260924 03:05:07.799502 13629 heartbeater.cc:464] Registering TS with master...
I20260924 03:05:07.799597 13629 heartbeater.cc:511] Master 127.0.98.62:39795 requested a full tablet report, sending...
W20260924 03:05:07.799932 13629 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 03:05:08 dist-test-slave-wgsb krb5kdc[12166](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790219108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:08.406991 13361 heartbeater.cc:464] Registering TS with master...
I20260924 03:05:08.407142 13361 heartbeater.cc:511] Master 127.0.98.62:39795 requested a full tablet report, sending...
I20260924 03:05:08.407567 13162 ts_manager.cc:194] Registered new tserver with Master: fd6fc90bdddf4ef5b4c2067105ae9287 (127.0.98.1:41019)
I20260924 03:05:08.408607 13162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41041
I20260924 03:05:08.600126 13495 heartbeater.cc:464] Registering TS with master...
I20260924 03:05:08.600287 13495 heartbeater.cc:511] Master 127.0.98.62:39795 requested a full tablet report, sending...
I20260924 03:05:08.600687 13162 ts_manager.cc:194] Registered new tserver with Master: e038465b23f44d66ba0400ddfbcae0c3 (127.0.98.2:46727)
I20260924 03:05:08.601619 13162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39091
Sep 24 03:05:08 dist-test-slave-wgsb krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 03:05:08.800113 13629 heartbeater.cc:464] Registering TS with master...
I20260924 03:05:08.800271 13629 heartbeater.cc:511] Master 127.0.98.62:39795 requested a full tablet report, sending...
I20260924 03:05:08.800549 13162 ts_manager.cc:194] Registered new tserver with Master: 985ebedc2f6c4f9389abd4c27309de48 (127.0.98.3:41107)
I20260924 03:05:08.801069 13162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41807
I20260924 03:05:08.811189 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:05:08.826233 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:08.815090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48204 (local address 127.0.98.62:39795)
0924 03:05:08.816256 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:08.816258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:08.816270 (+ 12us) server_negotiation.cc:407] Connection header received
0924 03:05:08.816301 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:08.816302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:08.816318 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:08.816403 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:08.817366 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:08.818180 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:08.819411 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:08.819754 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:08.819814 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:08.821372 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:08.821373 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:08.821374 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:08.821382 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:08.823145 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:08.824060 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:08.824061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:08.824062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:08.824129 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:08.824551 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:08.824552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:08.824553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:08.824696 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:08.824801 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:08.826015 (+ 1214us) server_negotiation.cc:299] Negotiation successful
0924 03:05:08.826121 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":78,"threads_started":1}
I20260924 03:05:08.828655 13162 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:05:08.830873 13162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:05:08.840606 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:08.834841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41019 (local address 127.0.0.1:51184)
0924 03:05:08.835979 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:08.835990 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:08.836048 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:08.836355 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:08.836356 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:08.836358 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:05:08.836400 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:08.836402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:08.837898 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:08.837899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:08.839012 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:08.839025 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:08.840309 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:08.840310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:08.840396 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:08.840412 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 03:05:08.840464 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 03:05:08.840522 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1122,"thread_start_us":62,"threads_started":1}
I20260924 03:05:08.841378 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:08.835122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51184 (local address 127.0.98.1:41019)
0924 03:05:08.836148 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:08.836152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:08.836161 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:05:08.836192 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:08.836194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:08.836208 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:08.836279 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:05:08.836479 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:08.837799 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:08.839129 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:08.840204 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:08.841151 (+ 947us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:08.841232 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:08.841260 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 03:05:08.841300 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":997,"thread_start_us":63,"threads_started":1}
I20260924 03:05:08.842314 13295 tablet_service.cc:1467] Processing CreateTablet for tablet fdbb659dd7ed43ffbd332e1aa69dc843 (DEFAULT_TABLE table=test-table [id=0230f2108d8147b3a2624a119894a2c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:05:08.842669 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdbb659dd7ed43ffbd332e1aa69dc843. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:05:08.844154 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:08.834614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46727 (local address 127.0.0.1:43834)
0924 03:05:08.835656 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:08.835664 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:08.835704 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:08.836109 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:08.836110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:08.836112 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:05:08.836138 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:08.836139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:08.837380 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:08.837381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:08.838761 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:08.838764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:08.843906 (+ 5142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:08.843907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:08.843975 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:08.843979 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:05:08.844037 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 03:05:08.844081 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1019,"thread_start_us":100,"threads_started":1}
I20260924 03:05:08.844501 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:08.835126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43834 (local address 127.0.98.2:46727)
0924 03:05:08.835353 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:08.835358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:08.835756 (+ 398us) server_negotiation.cc:407] Connection header received
0924 03:05:08.835792 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:08.835794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:08.835809 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:08.835874 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:05:08.836201 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:08.837285 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:08.838861 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:08.839633 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:08.839702 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:08.839778 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:08.844378 (+ 4600us) server_negotiation.cc:299] Negotiation successful
0924 03:05:08.844428 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":66,"threads_started":1}
I20260924 03:05:08.846098 13662 tablet_bootstrap.cc:492] T fdbb659dd7ed43ffbd332e1aa69dc843 P fd6fc90bdddf4ef5b4c2067105ae9287: Bootstrap starting.
I20260924 03:05:08.846810 13662 tablet_bootstrap.cc:654] T fdbb659dd7ed43ffbd332e1aa69dc843 P fd6fc90bdddf4ef5b4c2067105ae9287: Neither blocks nor log segments found. Creating new log.
I20260924 03:05:08.846836 13429 tablet_service.cc:1467] Processing CreateTablet for tablet fdbb659dd7ed43ffbd332e1aa69dc843 (DEFAULT_TABLE table=test-table [id=0230f2108d8147b3a2624a119894a2c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:05:08.847160 13429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdbb659dd7ed43ffbd332e1aa69dc843. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:05:08.847198 13662 log.cc:824] T fdbb659dd7ed43ffbd332e1aa69dc843 P fd6fc90bdddf4ef5b4c2067105ae9287: Log is configured to *not* fsync() on all Append() calls
I20260924 03:05:08.850559 13664 tablet_bootstrap.cc:492] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3: Bootstrap starting.
I20260924 03:05:08.850576 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:08.834888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40178 (local address 127.0.98.3:41107)
0924 03:05:08.835720 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:08.835724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:08.835736 (+ 12us) server_negotiation.cc:407] Connection header received
0924 03:05:08.835773 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:08.835775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:08.835790 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:08.835877 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:05:08.839821 (+ 3944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:08.843804 (+ 3983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:08.849011 (+ 5207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:08.849781 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:08.850126 (+ 345us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:08.850213 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:08.850448 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 03:05:08.850495 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":57,"threads_started":1}
I20260924 03:05:08.850813 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:08.834612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41107 (local address 127.0.0.1:40178)
0924 03:05:08.835423 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:08.835462 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:08.835525 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:08.835954 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:08.835957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:08.835959 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:05:08.836040 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:08.836042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:08.847789 (+ 11747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:08.847790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:08.848898 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:08.848902 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:08.850280 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:08.850281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:08.850354 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:08.850357 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:05:08.850688 (+ 331us) client_negotiation.cc:240] Negotiation successful
0924 03:05:08.850735 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"thread_start_us":110,"threads_started":1}
I20260924 03:05:08.851490 13664 tablet_bootstrap.cc:654] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3: Neither blocks nor log segments found. Creating new log.
I20260924 03:05:08.851672 13546 tablet_service.cc:1467] Processing CreateTablet for tablet fdbb659dd7ed43ffbd332e1aa69dc843 (DEFAULT_TABLE table=test-table [id=0230f2108d8147b3a2624a119894a2c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:05:08.851876 13664 log.cc:824] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3: Log is configured to *not* fsync() on all Append() calls
I20260924 03:05:08.852077 13546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdbb659dd7ed43ffbd332e1aa69dc843. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:05:08.852236 13662 tablet_bootstrap.cc:492] T fdbb659dd7ed43ffbd332e1aa69dc843 P fd6fc90bdddf4ef5b4c2067105ae9287: No bootstrap required, opened a new log
I20260924 03:05:08.852316 13662 ts_tablet_manager.cc:1397] T fdbb659dd7ed43ffbd332e1aa69dc843 P fd6fc90bdddf4ef5b4c2067105ae9287: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260924 03:05:08.852629 13664 tablet_bootstrap.cc:492] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3: No bootstrap required, opened a new log
I20260924 03:05:08.852684 13664 ts_tablet_manager.cc:1397] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:05:08.855294 13664 raft_consensus.cc:348] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985ebedc2f6c4f9389abd4c27309de48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41107 } } peers { permanent_uuid: "e038465b23f44d66ba0400ddfbcae0c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46727 } } peers { permanent_uuid: "fd6fc90bdddf4ef5b4c2067105ae9287" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41019 } }
I20260924 03:05:08.855461 13664 raft_consensus.cc:374] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:05:08.855345 13662 raft_consensus.cc:348] T fdbb659dd7ed43ffbd332e1aa69dc843 P fd6fc90bdddf4ef5b4c2067105ae9287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985ebedc2f6c4f9389abd4c27309de48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41107 } } peers { permanent_uuid: "e038465b23f44d66ba0400ddfbcae0c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46727 } } peers { permanent_uuid: "fd6fc90bdddf4ef5b4c2067105ae9287" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41019 } }
I20260924 03:05:08.855496 13664 raft_consensus.cc:729] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e038465b23f44d66ba0400ddfbcae0c3, State: Initialized, Role: FOLLOWER
I20260924 03:05:08.855505 13662 raft_consensus.cc:374] T fdbb659dd7ed43ffbd332e1aa69dc843 P fd6fc90bdddf4ef5b4c2067105ae9287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:05:08.855542 13662 raft_consensus.cc:729] T fdbb659dd7ed43ffbd332e1aa69dc843 P fd6fc90bdddf4ef5b4c2067105ae9287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd6fc90bdddf4ef5b4c2067105ae9287, State: Initialized, Role: FOLLOWER
I20260924 03:05:08.855708 13662 consensus_queue.cc:260] T fdbb659dd7ed43ffbd332e1aa69dc843 P fd6fc90bdddf4ef5b4c2067105ae9287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985ebedc2f6c4f9389abd4c27309de48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41107 } } peers { permanent_uuid: "e038465b23f44d66ba0400ddfbcae0c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46727 } } peers { permanent_uuid: "fd6fc90bdddf4ef5b4c2067105ae9287" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41019 } }
I20260924 03:05:08.855732 13664 consensus_queue.cc:260] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985ebedc2f6c4f9389abd4c27309de48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41107 } } peers { permanent_uuid: "e038465b23f44d66ba0400ddfbcae0c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46727 } } peers { permanent_uuid: "fd6fc90bdddf4ef5b4c2067105ae9287" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41019 } }
I20260924 03:05:08.855973 13662 ts_tablet_manager.cc:1428] T fdbb659dd7ed43ffbd332e1aa69dc843 P fd6fc90bdddf4ef5b4c2067105ae9287: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 03:05:08.856104 13361 heartbeater.cc:503] Master 127.0.98.62:39795 was elected leader, sending a full tablet report...
I20260924 03:05:08.857355 13664 ts_tablet_manager.cc:1428] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 03:05:08.857524 13495 heartbeater.cc:503] Master 127.0.98.62:39795 was elected leader, sending a full tablet report...
I20260924 03:05:08.859810 13666 tablet_bootstrap.cc:492] T fdbb659dd7ed43ffbd332e1aa69dc843 P 985ebedc2f6c4f9389abd4c27309de48: Bootstrap starting.
I20260924 03:05:08.860509 13666 tablet_bootstrap.cc:654] T fdbb659dd7ed43ffbd332e1aa69dc843 P 985ebedc2f6c4f9389abd4c27309de48: Neither blocks nor log segments found. Creating new log.
I20260924 03:05:08.860844 13666 log.cc:824] T fdbb659dd7ed43ffbd332e1aa69dc843 P 985ebedc2f6c4f9389abd4c27309de48: Log is configured to *not* fsync() on all Append() calls
I20260924 03:05:08.861683 13666 tablet_bootstrap.cc:492] T fdbb659dd7ed43ffbd332e1aa69dc843 P 985ebedc2f6c4f9389abd4c27309de48: No bootstrap required, opened a new log
I20260924 03:05:08.861757 13666 ts_tablet_manager.cc:1397] T fdbb659dd7ed43ffbd332e1aa69dc843 P 985ebedc2f6c4f9389abd4c27309de48: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:05:08.864338 13666 raft_consensus.cc:348] T fdbb659dd7ed43ffbd332e1aa69dc843 P 985ebedc2f6c4f9389abd4c27309de48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985ebedc2f6c4f9389abd4c27309de48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41107 } } peers { permanent_uuid: "e038465b23f44d66ba0400ddfbcae0c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46727 } } peers { permanent_uuid: "fd6fc90bdddf4ef5b4c2067105ae9287" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41019 } }
I20260924 03:05:08.864512 13666 raft_consensus.cc:374] T fdbb659dd7ed43ffbd332e1aa69dc843 P 985ebedc2f6c4f9389abd4c27309de48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:05:08.864547 13666 raft_consensus.cc:729] T fdbb659dd7ed43ffbd332e1aa69dc843 P 985ebedc2f6c4f9389abd4c27309de48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 985ebedc2f6c4f9389abd4c27309de48, State: Initialized, Role: FOLLOWER
I20260924 03:05:08.864699 13666 consensus_queue.cc:260] T fdbb659dd7ed43ffbd332e1aa69dc843 P 985ebedc2f6c4f9389abd4c27309de48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985ebedc2f6c4f9389abd4c27309de48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41107 } } peers { permanent_uuid: "e038465b23f44d66ba0400ddfbcae0c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46727 } } peers { permanent_uuid: "fd6fc90bdddf4ef5b4c2067105ae9287" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41019 } }
I20260924 03:05:08.864949 13666 ts_tablet_manager.cc:1428] T fdbb659dd7ed43ffbd332e1aa69dc843 P 985ebedc2f6c4f9389abd4c27309de48: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:05:08.864989 13629 heartbeater.cc:503] Master 127.0.98.62:39795 was elected leader, sending a full tablet report...
I20260924 03:05:08.873800 13668 raft_consensus.cc:482] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:05:08.873943 13668 raft_consensus.cc:504] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985ebedc2f6c4f9389abd4c27309de48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41107 } } peers { permanent_uuid: "e038465b23f44d66ba0400ddfbcae0c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46727 } } peers { permanent_uuid: "fd6fc90bdddf4ef5b4c2067105ae9287" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41019 } }
I20260924 03:05:08.874339 13668 leader_election.cc:288] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 985ebedc2f6c4f9389abd4c27309de48 (127.0.98.3:41107), fd6fc90bdddf4ef5b4c2067105ae9287 (127.0.98.1:41019)
I20260924 03:05:08.878520 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:08.874702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44553 (local address 127.0.98.3:41107)
0924 03:05:08.874746 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:08.874748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:08.874860 (+ 112us) server_negotiation.cc:407] Connection header received
0924 03:05:08.874927 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:08.874927 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:08.874939 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:08.874987 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:05:08.875220 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:08.875994 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:08.877531 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:08.878106 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:08.878157 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:08.878169 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:08.878389 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 03:05:08.878442 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:05:08.878545 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:08.874715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47303 (local address 127.0.98.1:41019)
0924 03:05:08.874791 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:08.874793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:08.874982 (+ 189us) server_negotiation.cc:407] Connection header received
0924 03:05:08.875086 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:08.875087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:08.875097 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:08.875124 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:05:08.875317 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:08.876065 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:08.877537 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:08.878148 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:08.878193 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:08.878202 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:08.878432 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 03:05:08.878474 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 03:05:08.878798 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:08.874525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41019 (local address 127.0.98.2:47303)
0924 03:05:08.874936 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:08.874943 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:08.875001 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:08.875184 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:08.875185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:08.875187 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:05:08.875251 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:08.875253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:08.876159 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:08.876159 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:08.877429 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:08.877432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:08.878279 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:08.878279 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:08.878355 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:08.878358 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:05:08.878657 (+ 299us) client_negotiation.cc:240] Negotiation successful
0924 03:05:08.878703 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":96,"threads_started":1}
I20260924 03:05:08.879195 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbb659dd7ed43ffbd332e1aa69dc843" candidate_uuid: "e038465b23f44d66ba0400ddfbcae0c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd6fc90bdddf4ef5b4c2067105ae9287" is_pre_election: true
I20260924 03:05:08.879350 13315 raft_consensus.cc:2454] T fdbb659dd7ed43ffbd332e1aa69dc843 P fd6fc90bdddf4ef5b4c2067105ae9287 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e038465b23f44d66ba0400ddfbcae0c3 in term 0.
I20260924 03:05:08.879458 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:08.874508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41107 (local address 127.0.98.2:44553)
0924 03:05:08.874782 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:08.874794 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:08.874855 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:08.875050 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:08.875051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:08.875053 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:05:08.875143 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:08.875145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:08.876107 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:08.876108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:08.877424 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:08.877427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:08.878234 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:08.878235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:08.878310 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:08.878314 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:05:08.879337 (+ 1023us) client_negotiation.cc:240] Negotiation successful
0924 03:05:08.879385 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":45,"thread_start_us":97,"threads_started":1}
I20260924 03:05:08.879558 13383 leader_election.cc:302] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e038465b23f44d66ba0400ddfbcae0c3, fd6fc90bdddf4ef5b4c2067105ae9287; no voters:
I20260924 03:05:08.879806 13567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbb659dd7ed43ffbd332e1aa69dc843" candidate_uuid: "e038465b23f44d66ba0400ddfbcae0c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "985ebedc2f6c4f9389abd4c27309de48" is_pre_election: true
I20260924 03:05:08.879932 13567 raft_consensus.cc:2454] T fdbb659dd7ed43ffbd332e1aa69dc843 P 985ebedc2f6c4f9389abd4c27309de48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e038465b23f44d66ba0400ddfbcae0c3 in term 0.
I20260924 03:05:08.880043 13668 raft_consensus.cc:2781] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:05:08.880095 13668 raft_consensus.cc:482] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:05:08.880119 13668 raft_consensus.cc:3037] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:05:08.880967 13668 raft_consensus.cc:504] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985ebedc2f6c4f9389abd4c27309de48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41107 } } peers { permanent_uuid: "e038465b23f44d66ba0400ddfbcae0c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46727 } } peers { permanent_uuid: "fd6fc90bdddf4ef5b4c2067105ae9287" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41019 } }
I20260924 03:05:08.881106 13668 leader_election.cc:288] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3 [CANDIDATE]: Term 1 election: Requested vote from peers 985ebedc2f6c4f9389abd4c27309de48 (127.0.98.3:41107), fd6fc90bdddf4ef5b4c2067105ae9287 (127.0.98.1:41019)
I20260924 03:05:08.881382 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbb659dd7ed43ffbd332e1aa69dc843" candidate_uuid: "e038465b23f44d66ba0400ddfbcae0c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd6fc90bdddf4ef5b4c2067105ae9287"
I20260924 03:05:08.881457 13315 raft_consensus.cc:3037] T fdbb659dd7ed43ffbd332e1aa69dc843 P fd6fc90bdddf4ef5b4c2067105ae9287 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:05:08.881449 13567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbb659dd7ed43ffbd332e1aa69dc843" candidate_uuid: "e038465b23f44d66ba0400ddfbcae0c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "985ebedc2f6c4f9389abd4c27309de48"
I20260924 03:05:08.881527 13567 raft_consensus.cc:3037] T fdbb659dd7ed43ffbd332e1aa69dc843 P 985ebedc2f6c4f9389abd4c27309de48 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:05:08.882366 13315 raft_consensus.cc:2454] T fdbb659dd7ed43ffbd332e1aa69dc843 P fd6fc90bdddf4ef5b4c2067105ae9287 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e038465b23f44d66ba0400ddfbcae0c3 in term 1.
I20260924 03:05:08.882375 13567 raft_consensus.cc:2454] T fdbb659dd7ed43ffbd332e1aa69dc843 P 985ebedc2f6c4f9389abd4c27309de48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e038465b23f44d66ba0400ddfbcae0c3 in term 1.
I20260924 03:05:08.882555 13383 leader_election.cc:302] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e038465b23f44d66ba0400ddfbcae0c3, fd6fc90bdddf4ef5b4c2067105ae9287; no voters:
I20260924 03:05:08.882648 13668 raft_consensus.cc:2781] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:05:08.882795 13668 raft_consensus.cc:686] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3 [term 1 LEADER]: Becoming Leader. State: Replica: e038465b23f44d66ba0400ddfbcae0c3, State: Running, Role: LEADER
I20260924 03:05:08.882882 13668 consensus_queue.cc:237] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985ebedc2f6c4f9389abd4c27309de48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41107 } } peers { permanent_uuid: "e038465b23f44d66ba0400ddfbcae0c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46727 } } peers { permanent_uuid: "fd6fc90bdddf4ef5b4c2067105ae9287" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41019 } }
I20260924 03:05:08.883829 13162 catalog_manager.cc:5495] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e038465b23f44d66ba0400ddfbcae0c3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e038465b23f44d66ba0400ddfbcae0c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985ebedc2f6c4f9389abd4c27309de48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e038465b23f44d66ba0400ddfbcae0c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd6fc90bdddf4ef5b4c2067105ae9287" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41019 } health_report { overall_health: UNKNOWN } } }
W20260924 03:05:08.887224 13362 tablet.cc:2369] T fdbb659dd7ed43ffbd332e1aa69dc843 P fd6fc90bdddf4ef5b4c2067105ae9287: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:05:08.897341 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:08.893822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43838 (local address 127.0.98.2:46727)
0924 03:05:08.894024 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:08.894027 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:08.894035 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:05:08.894075 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:08.894076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:08.894101 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:08.894166 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:05:08.894351 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:08.895044 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:08.896223 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:08.896526 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:08.896593 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:08.896797 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:05:08.896945 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:05:08.897046 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:08.897233 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 03:05:08.897258 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260924 03:05:08.899817 13315 raft_consensus.cc:1261] T fdbb659dd7ed43ffbd332e1aa69dc843 P fd6fc90bdddf4ef5b4c2067105ae9287 [term 1 FOLLOWER]: Refusing update from remote peer e038465b23f44d66ba0400ddfbcae0c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:05:08.899860 13567 raft_consensus.cc:1261] T fdbb659dd7ed43ffbd332e1aa69dc843 P 985ebedc2f6c4f9389abd4c27309de48 [term 1 FOLLOWER]: Refusing update from remote peer e038465b23f44d66ba0400ddfbcae0c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:05:08.900291 13673 consensus_queue.cc:1035] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd6fc90bdddf4ef5b4c2067105ae9287" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:05:08.900545 13668 consensus_queue.cc:1035] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "985ebedc2f6c4f9389abd4c27309de48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:05:08.905544 13162 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:05:08.905652 13162 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:05:08.906581 13162 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c90618d753114d59a04c2e7ad08475e3: Sending DeleteTablet for 3 replicas of tablet fdbb659dd7ed43ffbd332e1aa69dc843
I20260924 03:05:08.907045 13429 tablet_service.cc:1514] Processing DeleteTablet for tablet fdbb659dd7ed43ffbd332e1aa69dc843 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:05:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43834
I20260924 03:05:08.907244 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet fdbb659dd7ed43ffbd332e1aa69dc843 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:05:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51184
I20260924 03:05:08.907866 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 13200
I20260924 03:05:08.908269 13678 mvcc.cc:205] Tried to move back new op lower bound from 7332737470050246656 to 7332737469984694272. Current Snapshot: MvccSnapshot[applied={T|T < 7332737470050246656}]
I20260924 03:05:08.913081 13546 tablet_service.cc:1514] Processing DeleteTablet for tablet fdbb659dd7ed43ffbd332e1aa69dc843 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:05:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40178
I20260924 03:05:08.913578 13686 tablet_replica.cc:330] T fdbb659dd7ed43ffbd332e1aa69dc843 P 985ebedc2f6c4f9389abd4c27309de48: stopping tablet replica
I20260924 03:05:08.913774 13686 raft_consensus.cc:2229] T fdbb659dd7ed43ffbd332e1aa69dc843 P 985ebedc2f6c4f9389abd4c27309de48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:05:08.913932 13686 raft_consensus.cc:2258] T fdbb659dd7ed43ffbd332e1aa69dc843 P 985ebedc2f6c4f9389abd4c27309de48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:05:08.915719 13683 tablet_replica.cc:330] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3: stopping tablet replica
I20260924 03:05:08.915860 13683 raft_consensus.cc:2229] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:05:08.915987 13683 raft_consensus.cc:2258] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:05:08.917693 13686 ts_tablet_manager.cc:1905] T fdbb659dd7ed43ffbd332e1aa69dc843 P 985ebedc2f6c4f9389abd4c27309de48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:05:08.919842 13686 ts_tablet_manager.cc:1918] T fdbb659dd7ed43ffbd332e1aa69dc843 P 985ebedc2f6c4f9389abd4c27309de48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:05:08.919981 13686 log.cc:1196] T fdbb659dd7ed43ffbd332e1aa69dc843 P 985ebedc2f6c4f9389abd4c27309de48: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-2/wal/wals/fdbb659dd7ed43ffbd332e1aa69dc843
W20260924 03:05:08.920337 13383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41019 (error 108)
W20260924 03:05:08.920662 13147 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41019 (error 108)
I20260924 03:05:08.920838 13683 ts_tablet_manager.cc:1905] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:05:08.921283 13686 ts_tablet_manager.cc:1939] T fdbb659dd7ed43ffbd332e1aa69dc843 P 985ebedc2f6c4f9389abd4c27309de48: Deleting consensus metadata
I20260924 03:05:08.921782 13146 catalog_manager.cc:4841] TS 985ebedc2f6c4f9389abd4c27309de48 (127.0.98.3:41107): tablet fdbb659dd7ed43ffbd332e1aa69dc843 (table test-table [id=0230f2108d8147b3a2624a119894a2c3]) successfully deleted
I20260924 03:05:08.922263 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:08.922141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41019 (local address 127.0.98.2:41403)
0924 03:05:08.922193 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:08.922228 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 03:05:08.922892 13683 ts_tablet_manager.cc:1918] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:05:08.923045 13683 log.cc:1196] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218948286999-392-0/minicluster-data/ts-1/wal/wals/fdbb659dd7ed43ffbd332e1aa69dc843
I20260924 03:05:08.923233 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:08.923098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41019 (local address 127.0.0.1:51196)
0924 03:05:08.923151 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:08.923194 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 03:05:08.923514 13147 catalog_manager.cc:4584] TS fd6fc90bdddf4ef5b4c2067105ae9287 (127.0.98.1:41019): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fdbb659dd7ed43ffbd332e1aa69dc843: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41019: connect: Connection refused (error 111)
I20260924 03:05:08.923616 13683 ts_tablet_manager.cc:1939] T fdbb659dd7ed43ffbd332e1aa69dc843 P e038465b23f44d66ba0400ddfbcae0c3: Deleting consensus metadata
I20260924 03:05:08.924082 13147 catalog_manager.cc:4841] TS e038465b23f44d66ba0400ddfbcae0c3 (127.0.98.2:46727): tablet fdbb659dd7ed43ffbd332e1aa69dc843 (table test-table [id=0230f2108d8147b3a2624a119894a2c3]) successfully deleted
I20260924 03:05:08.924111 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 13365
I20260924 03:05:08.930853 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 13498
I20260924 03:05:08.937399 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 13129
I20260924 03:05:08.944106 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 03:05:09.289179 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 03:05:09.289283 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 03:05:09.634845 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 03:05:09.635 UTC [12196] LOG: received smart shutdown request
2026-09-24 03:05:09.638 UTC [12196] LOG: background worker "logical replication launcher" (PID 12205) exited with exit code 1
2026-09-24 03:05:09.638 UTC [12200] LOG: shutting down
2026-09-24 03:05:09.837 UTC [12196] LOG: database system is shut down
2026-09-24T03:05:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53838 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:05:10 dist-test-slave-wgsb krb5kdc[13697](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:05:10 dist-test-slave-wgsb krb5kdc[13697](info): set up 2 sockets
Sep 24 03:05:10 dist-test-slave-wgsb krb5kdc[13697](info): commencing operation
krb5kdc: starting...
W20260924 03:05:11.015642 392 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:05:11 dist-test-slave-wgsb krb5kdc[13697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:05:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:05:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:05:11.139020 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36893
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36893,127.0.98.61:43357,127.0.98.60:38519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:05:11.225276 13710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:11.225502 13710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:11.225528 13710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:11.226691 13710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:05:11.226730 13710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:11.226744 13710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:11.226758 13710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:05:11.226769 13710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:05:11.228344 13710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36893,127.0.98.61:43357,127.0.98.60:38519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:11.228554 13710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:11.228736 13710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:11.233670 13716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:11.233719 13715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:11.233670 13718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:11.233934 13710 server_base.cc:1034] running on GCE node
I20260924 03:05:11.234232 13710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:11.234496 13710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:11.235723 13710 hybrid_clock.cc:648] HybridClock initialized: now 1790219111235708 us; error 27 us; skew 500 ppm
Sep 24 03:05:11 dist-test-slave-wgsb krb5kdc[13697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:11.247900 13710 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:05:11.248544 13710 webserver.cc:460] Webserver started at http://127.0.98.62:34085/ using document root <none> and password file <none>
I20260924 03:05:11.248814 13710 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:11.248888 13710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:11.249051 13710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:05:11.250186 13710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "0d0c31da3af741dd959e4b4d8b89e9e7"
format_stamp: "Formatted at 2026-09-24 03:05:11 on dist-test-slave-wgsb"
server_key: "d816bdfa4a7f0e32fb868f7584b1f630"
server_key_iv: "9d07cffe53d51cdf6d224d932a98c001"
server_key_version: "encryptionkey@0"
I20260924 03:05:11.250550 13710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "0d0c31da3af741dd959e4b4d8b89e9e7"
format_stamp: "Formatted at 2026-09-24 03:05:11 on dist-test-slave-wgsb"
server_key: "d816bdfa4a7f0e32fb868f7584b1f630"
server_key_iv: "9d07cffe53d51cdf6d224d932a98c001"
server_key_version: "encryptionkey@0"
I20260924 03:05:11.252025 13710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:05:11.252772 13725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:11.252956 13710 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:05:11.253113 13710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "0d0c31da3af741dd959e4b4d8b89e9e7"
format_stamp: "Formatted at 2026-09-24 03:05:11 on dist-test-slave-wgsb"
server_key: "d816bdfa4a7f0e32fb868f7584b1f630"
server_key_iv: "9d07cffe53d51cdf6d224d932a98c001"
server_key_version: "encryptionkey@0"
I20260924 03:05:11.253188 13710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:11.285143 13710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:11.287181 13710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:11.287359 13710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:11.291760 13710 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36893
I20260924 03:05:11.291962 13777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36893 every 8 connection(s)
I20260924 03:05:11.292224 13710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:05:11.293314 13778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:05:11.294896 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 13710
I20260924 03:05:11.295022 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:05:11.295198 392 external_mini_cluster.cc:1428] Setting key f23c97d060552418d1aca55fae9bdc1a
I20260924 03:05:11.295056 13778 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36893 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:05:11 dist-test-slave-wgsb krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260924 03:05:11.307320 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.296253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59934 (local address 127.0.98.62:36893)
0924 03:05:11.296403 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:11.296409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:11.296450 (+ 41us) server_negotiation.cc:407] Connection header received
0924 03:05:11.297165 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:11.297168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:11.297186 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:11.297271 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:11.298155 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.299541 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.300467 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.301035 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.301155 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:11.303629 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:11.303638 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:11.303638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:11.303647 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:11.305157 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.305908 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.305911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.305911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.306012 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.306262 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.306263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.306263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.306562 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:11.306664 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:11.306987 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 03:05:11.307158 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
I20260924 03:05:11.307346 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.296017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36893 (local address 127.0.0.1:59934)
0924 03:05:11.296222 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:11.296267 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:11.296982 (+ 715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:11.297309 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:11.297313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:11.297729 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:11.297948 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:11.297953 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.299679 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.299680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:11.300375 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:11.300377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.301121 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.301122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:11.301180 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:11.301786 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:11.301794 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:11.303522 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:11.305232 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:11.305237 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:11.305239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:11.305718 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:11.306091 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:11.306092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:11.306092 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:11.306182 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:11.306674 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:11.306682 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:11.306772 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 03:05:11.306869 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 03:05:11.307091 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":117,"threads_started":1}
I20260924 03:05:11.308198 13778 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43357 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:05:11.308610 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.308481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43357 (local address 127.0.0.1:48956)
0924 03:05:11.308536 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:11.308589 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 03:05:11.308751 13727 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43357: connect: Connection refused (error 111)
I20260924 03:05:11.309221 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.309119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43357 (local address 127.0.0.1:48970)
0924 03:05:11.309171 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:11.309197 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 03:05:11.309334 13778 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43357: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43357: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 03:05:11.323835 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43357
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36893,127.0.98.61:43357,127.0.98.60:38519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:05:11.358559 13778 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43357 } attempt: 1
I20260924 03:05:11.359093 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.358910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43357 (local address 127.0.0.1:48978)
0924 03:05:11.359026 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:11.359062 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 03:05:11.359508 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.359421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43357 (local address 127.0.0.1:48994)
0924 03:05:11.359462 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:11.359487 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 03:05:11.359644 13778 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43357: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43357: connect: Connection refused (error 111)
W20260924 03:05:11.418186 13785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:11.418447 13785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:11.418499 13785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:11.420713 13785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:05:11.422143 13785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:11.422197 13785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:11.422227 13785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:05:11.422241 13785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:05:11.424768 13785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36893,127.0.98.61:43357,127.0.98.60:38519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:11.426436 13785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:11.426987 13785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:05:11.427810 13778 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43357 } attempt: 2
I20260924 03:05:11.429057 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.428492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43357 (local address 127.0.0.1:49010)
0924 03:05:11.428699 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:11.428752 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
I20260924 03:05:11.430022 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.429806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43357 (local address 127.0.0.1:49022)
0924 03:05:11.429900 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:11.429987 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260924 03:05:11.430184 13778 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43357: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43357: connect: Connection refused (error 111)
W20260924 03:05:11.435710 13790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:11.435843 13793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:11.435849 13785 server_base.cc:1034] running on GCE node
W20260924 03:05:11.435710 13791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:11.436416 13785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:11.436749 13785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:11.437927 13785 hybrid_clock.cc:648] HybridClock initialized: now 1790219111437890 us; error 63 us; skew 500 ppm
Sep 24 03:05:11 dist-test-slave-wgsb krb5kdc[13697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:11.444063 13785 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 03:05:11.444784 13785 webserver.cc:460] Webserver started at http://127.0.98.61:44869/ using document root <none> and password file <none>
I20260924 03:05:11.445013 13785 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:11.445107 13785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:11.445276 13785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:05:11.447096 13785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-1/data/instance:
uuid: "484745f9cada4ce0ab1f81bef771eed8"
format_stamp: "Formatted at 2026-09-24 03:05:11 on dist-test-slave-wgsb"
server_key: "34777cba47f82001cc88a1330159efc3"
server_key_iv: "bf24906b97082f5432007727e9207119"
server_key_version: "encryptionkey@0"
I20260924 03:05:11.447939 13785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-1/wal/instance:
uuid: "484745f9cada4ce0ab1f81bef771eed8"
format_stamp: "Formatted at 2026-09-24 03:05:11 on dist-test-slave-wgsb"
server_key: "34777cba47f82001cc88a1330159efc3"
server_key_iv: "bf24906b97082f5432007727e9207119"
server_key_version: "encryptionkey@0"
I20260924 03:05:11.449790 13785 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260924 03:05:11.452119 13800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:11.452479 13785 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:05:11.452708 13785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-1/wal
uuid: "484745f9cada4ce0ab1f81bef771eed8"
format_stamp: "Formatted at 2026-09-24 03:05:11 on dist-test-slave-wgsb"
server_key: "34777cba47f82001cc88a1330159efc3"
server_key_iv: "bf24906b97082f5432007727e9207119"
server_key_version: "encryptionkey@0"
I20260924 03:05:11.452814 13785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:11.475194 13785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:11.478186 13785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:11.478354 13785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:11.487810 13785 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43357
I20260924 03:05:11.488341 13785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-1/data/info.pb
I20260924 03:05:11.488708 13852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43357 every 8 connection(s)
I20260924 03:05:11.489836 13853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:05:11.491255 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 13785
I20260924 03:05:11.491495 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-1/wal/instance
I20260924 03:05:11.491745 392 external_mini_cluster.cc:1428] Setting key 1e5d56906dd20a2be6a28b192b73c5e9
I20260924 03:05:11.491709 13853 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36893 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 03:05:11 dist-test-slave-wgsb krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:05:11.509598 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.492642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59946 (local address 127.0.98.62:36893)
0924 03:05:11.492721 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:11.492724 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:11.493404 (+ 680us) server_negotiation.cc:407] Connection header received
0924 03:05:11.493977 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:11.493979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:11.493996 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:11.494059 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:11.494783 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.495400 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.497242 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.497491 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.497592 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:11.505756 (+ 8164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:11.505758 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:11.505759 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:11.505770 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:11.507740 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.508242 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.508244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.508245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.508319 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.508638 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.508640 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.508641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.508924 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:11.509025 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:11.509347 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 03:05:11.509459 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 03:05:11.509706 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.492608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36893 (local address 127.0.0.1:59946)
0924 03:05:11.493247 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:11.493290 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:11.493856 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:11.494175 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:11.494178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:11.494456 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:11.494673 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:11.494677 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.495506 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.495508 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:11.497011 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:11.497014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.497601 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.497603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:11.497742 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:11.498350 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:11.498360 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:11.505604 (+ 7244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:11.507849 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:11.507858 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:11.507862 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:11.508152 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:11.508394 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:11.508395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:11.508396 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:11.508526 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:11.509028 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:11.509037 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:11.509188 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 03:05:11.509297 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 03:05:11.509442 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":106,"threads_started":1}
I20260924 03:05:11.510370 13853 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43357 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:05:11 dist-test-slave-wgsb krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:05:11.520275 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.510691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43357 (local address 127.0.0.1:49024)
0924 03:05:11.510928 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:11.510939 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:11.511001 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:11.511190 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:11.511191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:11.511360 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:11.511449 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:11.511451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.512462 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.512464 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:11.513490 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:11.513493 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.513942 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.513943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:11.514004 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:11.514481 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:11.514485 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:11.516792 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:11.518905 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:11.518906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:11.518906 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:11.519086 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:11.519387 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:11.519388 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:11.519389 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:11.519526 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:11.519920 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:11.519921 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:11.519966 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 03:05:11.520051 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 03:05:11.520134 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218}
I20260924 03:05:11.520537 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.510774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49024 (local address 127.0.98.61:43357)
0924 03:05:11.510943 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:11.510947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:11.510990 (+ 43us) server_negotiation.cc:407] Connection header received
0924 03:05:11.511083 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:11.511085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:11.511098 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:11.511166 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:11.511577 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.512374 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.513574 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.513874 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.513934 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:11.516892 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:11.516903 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:11.516903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:11.516910 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:11.518805 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.519180 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.519184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.519184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.519291 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.519653 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.519654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.519655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.519828 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:11.519897 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:11.520291 (+ 394us) server_negotiation.cc:299] Negotiation successful
0924 03:05:11.520388 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":93,"threads_started":1}
I20260924 03:05:11.521283 13853 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:05:11.521356 13778 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43357 } attempt: 3
I20260924 03:05:11.521953 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.521562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38519 (local address 127.0.0.1:56294)
0924 03:05:11.521790 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:11.521931 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 03:05:11.522125 13801 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38519: connect: Connection refused (error 111)
I20260924 03:05:11.522606 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.522503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38519 (local address 127.0.0.1:56302)
0924 03:05:11.522554 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:11.522584 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 03:05:11.522751 13853 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38519: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38519: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 03:05:11.524945 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38519
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36893,127.0.98.61:43357,127.0.98.60:38519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:05:11 dist-test-slave-wgsb krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:05:11.529348 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.521672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49040 (local address 127.0.98.61:43357)
0924 03:05:11.521733 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:11.521735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:11.521745 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:05:11.521765 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:11.521765 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:11.521774 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:11.521822 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:11.522174 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.522914 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.523912 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.524143 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.524210 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:11.526569 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:11.526571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:11.526571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:11.526580 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:11.528074 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.528525 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.528526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.528526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.528605 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.528844 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.528845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.528845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.528946 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:11.529033 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:11.529224 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 03:05:11.529288 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 03:05:11.529493 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.521596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43357 (local address 127.0.0.1:49040)
0924 03:05:11.521645 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:11.521658 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:11.521715 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:11.521843 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:11.521844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:11.521994 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:11.522086 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:11.522088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.522999 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.523000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:11.523782 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:11.523784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.524228 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.524230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:11.524321 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:11.524776 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:11.524782 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:11.526470 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:11.528161 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:11.528162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:11.528163 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:11.528443 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:11.528686 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:11.528687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:11.528688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:11.528769 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:11.529050 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:11.529051 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:11.529111 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 03:05:11.529209 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 03:05:11.529339 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 03:05:11.530103 13778 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:05:11.530629 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.530385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38519 (local address 127.0.0.1:56306)
0924 03:05:11.530549 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:11.530588 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260924 03:05:11.531114 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.530956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38519 (local address 127.0.0.1:56322)
0924 03:05:11.531048 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:11.531082 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260924 03:05:11.531241 13778 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38519: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38519: connect: Connection refused (error 111)
I20260924 03:05:11.562410 13778 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } attempt: 1
I20260924 03:05:11.562938 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.562778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38519 (local address 127.0.0.1:56334)
0924 03:05:11.562847 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:11.562900 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 03:05:11.563508 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.563367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38519 (local address 127.0.0.1:56336)
0924 03:05:11.563435 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:11.563474 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 03:05:11.563675 13778 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38519: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38519: connect: Connection refused (error 111)
I20260924 03:05:11.572086 13853 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } attempt: 1
I20260924 03:05:11.572574 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.572447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38519 (local address 127.0.0.1:56348)
0924 03:05:11.572505 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:11.572544 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 03:05:11.572933 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.572844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38519 (local address 127.0.0.1:56354)
0924 03:05:11.572888 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:11.572914 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 03:05:11.573027 13853 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38519: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38519: connect: Connection refused (error 111)
W20260924 03:05:11.614089 13860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:11.614326 13860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:11.614357 13860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:11.615702 13860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:05:11.615747 13860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:11.615762 13860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:11.615775 13860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:05:11.615787 13860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:05:11.617451 13860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36893,127.0.98.61:43357,127.0.98.60:38519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:11.617722 13860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:11.617962 13860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:11.623471 13868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:11.623520 13860 server_base.cc:1034] running on GCE node
W20260924 03:05:11.623482 13866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:11.623522 13865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:11.624076 13860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:11.624401 13860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:11.625619 13860 hybrid_clock.cc:648] HybridClock initialized: now 1790219111625597 us; error 39 us; skew 500 ppm
Sep 24 03:05:11 dist-test-slave-wgsb krb5kdc[13697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:11.628576 13860 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 03:05:11.629110 13860 webserver.cc:460] Webserver started at http://127.0.98.60:39003/ using document root <none> and password file <none>
I20260924 03:05:11.629379 13860 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:11.629455 13860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:11.629619 13860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:05:11.630807 13860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-2/data/instance:
uuid: "171d7dcf3c3a4ab084f5ad999ad003a1"
format_stamp: "Formatted at 2026-09-24 03:05:11 on dist-test-slave-wgsb"
server_key: "756d0132d5e1c79a79d5653531f0599e"
server_key_iv: "98138b0acbeb7ecdb36d491eeb6d3988"
server_key_version: "encryptionkey@0"
I20260924 03:05:11.631214 13860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-2/wal/instance:
uuid: "171d7dcf3c3a4ab084f5ad999ad003a1"
format_stamp: "Formatted at 2026-09-24 03:05:11 on dist-test-slave-wgsb"
server_key: "756d0132d5e1c79a79d5653531f0599e"
server_key_iv: "98138b0acbeb7ecdb36d491eeb6d3988"
server_key_version: "encryptionkey@0"
I20260924 03:05:11.632740 13860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:05:11.633636 13875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:11.633829 13860 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:05:11.633942 13860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-2/wal
uuid: "171d7dcf3c3a4ab084f5ad999ad003a1"
format_stamp: "Formatted at 2026-09-24 03:05:11 on dist-test-slave-wgsb"
server_key: "756d0132d5e1c79a79d5653531f0599e"
server_key_iv: "98138b0acbeb7ecdb36d491eeb6d3988"
server_key_version: "encryptionkey@0"
I20260924 03:05:11.634037 13860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:11.638855 13778 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } attempt: 2
I20260924 03:05:11.639284 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.639157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38519 (local address 127.0.0.1:56364)
0924 03:05:11.639218 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:11.639256 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 03:05:11.639706 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.639550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38519 (local address 127.0.0.1:56376)
0924 03:05:11.639635 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:11.639675 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 03:05:11.639837 13778 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38519: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38519: connect: Connection refused (error 111)
I20260924 03:05:11.641189 13853 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } attempt: 2
I20260924 03:05:11.641588 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.641455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38519 (local address 127.0.0.1:56386)
0924 03:05:11.641517 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:11.641555 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 03:05:11.642089 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.641997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38519 (local address 127.0.0.1:56388)
0924 03:05:11.642047 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:11.642071 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 03:05:11.642237 13853 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38519: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38519: connect: Connection refused (error 111)
I20260924 03:05:11.644709 13860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:11.647471 13860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:11.647651 13860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:11.652081 13860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38519
I20260924 03:05:11.652148 13927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38519 every 8 connection(s)
I20260924 03:05:11.652549 13860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-2/data/info.pb
I20260924 03:05:11.653723 13928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:05:11.655346 13928 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36893 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:05:11.660562 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 13860
I20260924 03:05:11.660661 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218948286999-392-0/minicluster-data/master-2/wal/instance
I20260924 03:05:11.660851 392 external_mini_cluster.cc:1428] Setting key 5f472b18ffcbedb053ff4f1f1bda73b4
Sep 24 03:05:11 dist-test-slave-wgsb krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 03:05:11 dist-test-slave-wgsb krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 03:05:11.666131 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.656187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59962 (local address 127.0.98.62:36893)
0924 03:05:11.656321 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:11.656323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:11.656567 (+ 244us) server_negotiation.cc:407] Connection header received
0924 03:05:11.657169 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:11.657171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:11.657183 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:11.657240 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:11.657916 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.658406 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.659440 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.659705 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.659792 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:11.662150 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:11.662152 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:11.662153 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:11.662161 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:11.663722 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.664168 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.664170 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.664170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.664231 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.664940 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.664942 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.664943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.665169 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:11.665506 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:11.665891 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 03:05:11.666011 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260924 03:05:11.666289 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.656143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36893 (local address 127.0.0.1:59962)
0924 03:05:11.656407 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:11.656474 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:11.657060 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:11.657320 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:11.657323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:11.657582 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:11.657827 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:11.657833 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.658501 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.658502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:11.659347 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:11.659349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.659801 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.659802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:11.659942 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:11.660483 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:11.660492 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:11.662056 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:11.663823 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:11.663837 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:11.663839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:11.664075 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:11.664300 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:11.664301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:11.664301 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:11.664433 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:11.665468 (+ 1035us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:11.665478 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:11.665680 (+ 202us) client_negotiation.cc:769] Sending connection context
0924 03:05:11.665831 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 03:05:11.666018 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":100,"threads_started":1}
I20260924 03:05:11.666965 13928 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43357 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:05:11.669948 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.661225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56390 (local address 127.0.98.60:38519)
0924 03:05:11.661555 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:11.661559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:11.661574 (+ 15us) server_negotiation.cc:407] Connection header received
0924 03:05:11.661591 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:11.661593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:11.661609 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:11.661695 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:11.662281 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.662979 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.663821 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.664148 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.664242 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:11.666555 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:11.666562 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:11.666563 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:11.666572 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:11.668637 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.669067 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.669071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.669071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.669135 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.669339 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.669340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.669340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.669559 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:11.669707 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:11.669812 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 03:05:11.669875 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":115,"threads_started":1}
Sep 24 03:05:11 dist-test-slave-wgsb krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 03:05:11 dist-test-slave-wgsb krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:05:11.674962 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.667235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43357 (local address 127.0.0.1:49046)
0924 03:05:11.667666 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:11.667678 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:11.667727 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:11.667852 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:11.667853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:11.668060 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:11.668107 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:11.668109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.668818 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.668819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:11.669498 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:11.669500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.670164 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.670165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:11.670233 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:11.670748 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:11.670753 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:11.672168 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:11.673948 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:11.673949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:11.673949 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:11.674103 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:11.674317 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:11.674318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:11.674318 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:11.674381 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:11.674638 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:11.674639 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:11.674680 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 03:05:11.674764 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 03:05:11.674865 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412}
I20260924 03:05:11.674962 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.667312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49046 (local address 127.0.98.61:43357)
0924 03:05:11.667363 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:11.667366 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:11.667727 (+ 361us) server_negotiation.cc:407] Connection header received
0924 03:05:11.667767 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:11.667768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:11.667781 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:11.667865 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:11.668188 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.668729 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.669583 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.670025 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.670105 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:11.672261 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:11.672262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:11.672263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:11.672271 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:11.673771 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.674193 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.674195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.674195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.674250 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.674454 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.674455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.674455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.674563 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:11.674632 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:11.674790 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 03:05:11.674864 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 03:05:11.675262 13928 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:05:11.678007 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.670870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59972 (local address 127.0.98.62:36893)
0924 03:05:11.670915 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:11.670917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:11.670925 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:05:11.670949 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:11.670950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:11.670959 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:11.671085 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:11.671570 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.672137 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.672824 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.673095 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.673177 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:11.675331 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:11.675332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:11.675332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:11.675340 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:11.676814 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.677169 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.677170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.677170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.677218 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.677432 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.677433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.677433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.677571 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:11.677773 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:11.677813 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 03:05:11.677889 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
Sep 24 03:05:11 dist-test-slave-wgsb krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 03:05:11.682276 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.675564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38519 (local address 127.0.0.1:56402)
0924 03:05:11.675628 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:11.675638 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:11.675675 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:11.675783 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:11.675784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:11.675925 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:11.675954 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:11.675955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.676603 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.676604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:11.677283 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:11.677286 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.677762 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.677763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:11.677821 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:11.678199 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:11.678204 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:11.679578 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:11.681270 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:11.681270 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:11.681271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:11.681529 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:11.681734 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:11.681735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:11.681735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:11.681787 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:11.682067 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:11.682067 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:11.682101 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 03:05:11.682150 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 03:05:11.682215 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 03:05:11.682487 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.675641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56402 (local address 127.0.98.60:38519)
0924 03:05:11.675689 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:11.675691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:11.675698 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:05:11.675718 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:11.675718 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:11.675735 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:11.675777 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:11.676037 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.676516 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.677381 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.677691 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.677756 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:11.679693 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:11.679694 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:11.679694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:11.679700 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:11.681202 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.681598 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.681599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.681599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.681661 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.681851 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.681852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.681852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.681994 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:11.682072 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:11.682200 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 03:05:11.682285 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Sep 24 03:05:11 dist-test-slave-wgsb krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:05:11.684122 13928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1: Bootstrap starting.
I20260924 03:05:11.685232 13928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1: Neither blocks nor log segments found. Creating new log.
I20260924 03:05:11.685567 13928 log.cc:824] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1: Log is configured to *not* fsync() on all Append() calls
I20260924 03:05:11.686462 13928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1: No bootstrap required, opened a new log
I20260924 03:05:11.686851 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.678629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49058 (local address 127.0.98.61:43357)
0924 03:05:11.678663 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:11.678665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:11.678672 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:05:11.678705 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:11.678706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:11.678714 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:11.678772 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:11.679324 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.679955 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.680955 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.681133 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.681188 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:11.683900 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:11.683901 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:11.683901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:11.683908 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:11.685636 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.686018 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.686019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.686020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.686073 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.686288 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.686289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.686289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.686418 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:11.686564 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:11.686676 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 03:05:11.686752 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:05:11.688614 13928 raft_consensus.cc:348] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0c31da3af741dd959e4b4d8b89e9e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36893 } } peers { permanent_uuid: "484745f9cada4ce0ab1f81bef771eed8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43357 } } peers { permanent_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } }
I20260924 03:05:11.688791 13928 raft_consensus.cc:374] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:05:11.688830 13928 raft_consensus.cc:729] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171d7dcf3c3a4ab084f5ad999ad003a1, State: Initialized, Role: FOLLOWER
I20260924 03:05:11.688963 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:05:11.688959 13928 consensus_queue.cc:260] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0c31da3af741dd959e4b4d8b89e9e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36893 } } peers { permanent_uuid: "484745f9cada4ce0ab1f81bef771eed8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43357 } } peers { permanent_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } }
I20260924 03:05:11.689287 13928 sys_catalog.cc:564] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:05:11.689247 13934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0c31da3af741dd959e4b4d8b89e9e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36893 } } peers { permanent_uuid: "484745f9cada4ce0ab1f81bef771eed8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43357 } } peers { permanent_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } } }
I20260924 03:05:11.689378 13934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 03:05:11.691505 13951 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:05:11.691565 13951 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:05:11.691540 13860 master_runner.cc:186] Error getting master registration for 127.0.98.62:36893: OK, instance_id { permanent_uuid: "0d0c31da3af741dd959e4b4d8b89e9e7" instance_seqno: 1790219111290458 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:05:11.694557 13860 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:05:11.697947 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.690536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59980 (local address 127.0.98.62:36893)
0924 03:05:11.690635 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:11.690638 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:11.691192 (+ 554us) server_negotiation.cc:407] Connection header received
0924 03:05:11.691293 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:11.691294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:11.691307 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:11.691338 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:11.692106 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.692549 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.693265 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.693454 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.693540 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:11.695032 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:11.695034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:11.695034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:11.695042 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:11.696728 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.697123 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.697125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.697125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.697180 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.697374 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.697374 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.697375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.697497 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:11.697574 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:11.697769 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 03:05:11.697848 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 03:05:11.698787 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.690582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49072 (local address 127.0.98.61:43357)
0924 03:05:11.690672 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:11.690674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:11.690682 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:05:11.690732 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:11.690733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:11.690743 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:11.690804 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:11.691500 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.692232 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.693325 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.693636 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.693680 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:11.695668 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:11.695670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:11.695670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:11.695678 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:11.697478 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.697914 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.697915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.697916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.697971 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.698212 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.698213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.698214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.698349 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:11.698416 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:11.698622 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 03:05:11.698699 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 03:05:11.703229 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.690697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56414 (local address 127.0.98.60:38519)
0924 03:05:11.690736 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:11.690738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:11.694738 (+ 4000us) server_negotiation.cc:407] Connection header received
0924 03:05:11.694852 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:11.694853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:11.694865 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:11.694897 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:11.695640 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.697142 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.698142 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.698427 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.698487 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:11.700284 (+ 1797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:11.700286 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:11.700286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:11.700294 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:11.701900 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.702290 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.702291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.702291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.702352 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.702566 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.702567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.702567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.702740 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:11.702838 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:11.703048 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 03:05:11.703130 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:05:11.733395 13853 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } attempt: 3
Sep 24 03:05:11 dist-test-slave-wgsb krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 03:05:11.738986 13778 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } attempt: 3
I20260924 03:05:11.741900 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.733890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56422 (local address 127.0.98.60:38519)
0924 03:05:11.733962 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:11.733964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:11.733992 (+ 28us) server_negotiation.cc:407] Connection header received
0924 03:05:11.734100 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:11.734101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:11.734111 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:11.734186 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:11.734524 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.735246 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.736367 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.736674 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.736743 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:11.738768 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:11.738769 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:11.738769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:11.738777 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:11.740694 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.741086 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.741087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.741088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.741148 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.741354 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.741355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.741355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.741486 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:11.741563 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:11.741726 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 03:05:11.741806 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 03:05:11.741900 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.733825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38519 (local address 127.0.0.1:56422)
0924 03:05:11.733933 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:11.733948 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:11.734009 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:11.734191 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:11.734192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:11.734384 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:11.734430 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:11.734432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.735331 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.735332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:11.736216 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:11.736220 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.736760 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.736760 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:11.736823 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:11.737281 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:11.737287 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:11.738677 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:11.740777 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:11.740778 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:11.740778 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:11.741008 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:11.741219 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:11.741220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:11.741220 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:11.741283 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:11.741566 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:11.741567 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:11.741615 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 03:05:11.741687 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 03:05:11.741791 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
Sep 24 03:05:11 dist-test-slave-wgsb krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 03:05:11.743750 13853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8: Bootstrap starting.
I20260924 03:05:11.744483 13853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8: Neither blocks nor log segments found. Creating new log.
I20260924 03:05:11.744800 13853 log.cc:824] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8: Log is configured to *not* fsync() on all Append() calls
I20260924 03:05:11.745771 13853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8: No bootstrap required, opened a new log
I20260924 03:05:11.747095 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.739230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38519 (local address 127.0.0.1:56432)
0924 03:05:11.739277 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:11.739287 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:11.739322 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:11.739646 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:11.739647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:11.739810 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:11.739856 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:11.739858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.740590 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.740591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:11.741360 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:11.741362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.741854 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.741856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:11.741926 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:11.742346 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:11.742351 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:11.743539 (+ 1188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:11.746074 (+ 2535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:11.746075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:11.746075 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:11.746244 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:11.746452 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:11.746453 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:11.746453 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:11.746514 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:11.746775 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:11.746776 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:11.746820 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 03:05:11.746900 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 03:05:11.746998 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 03:05:11.747129 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:11.739298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56432 (local address 127.0.98.60:38519)
0924 03:05:11.739488 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:11.739489 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:11.739497 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:05:11.739518 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:11.739519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:11.739529 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:11.739642 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:11.739965 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.740484 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.741436 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:11.741768 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:11.742801 (+ 1033us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:11.743626 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:11.743627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:11.743628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:11.743634 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:11.745986 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.746315 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.746316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.746317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.746374 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:11.746575 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:11.746576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:11.746576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:11.746700 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:11.746771 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:11.746956 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 03:05:11.747029 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":125,"threads_started":1}
I20260924 03:05:11.747797 13853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0c31da3af741dd959e4b4d8b89e9e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36893 } } peers { permanent_uuid: "484745f9cada4ce0ab1f81bef771eed8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43357 } } peers { permanent_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } }
I20260924 03:05:11.747936 13853 raft_consensus.cc:374] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:05:11.747972 13853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 484745f9cada4ce0ab1f81bef771eed8, State: Initialized, Role: FOLLOWER
I20260924 03:05:11.748090 13853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0c31da3af741dd959e4b4d8b89e9e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36893 } } peers { permanent_uuid: "484745f9cada4ce0ab1f81bef771eed8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43357 } } peers { permanent_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } }
I20260924 03:05:11.748389 13853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:05:11.748858 13778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d0c31da3af741dd959e4b4d8b89e9e7: Bootstrap starting.
I20260924 03:05:11.749574 13778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d0c31da3af741dd959e4b4d8b89e9e7: Neither blocks nor log segments found. Creating new log.
I20260924 03:05:11.748342 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0c31da3af741dd959e4b4d8b89e9e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36893 } } peers { permanent_uuid: "484745f9cada4ce0ab1f81bef771eed8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43357 } } peers { permanent_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } } }
I20260924 03:05:11.749727 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:05:11.749917 13778 log.cc:824] T 00000000000000000000000000000000 P 0d0c31da3af741dd959e4b4d8b89e9e7: Log is configured to *not* fsync() on all Append() calls
I20260924 03:05:11.750686 13778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d0c31da3af741dd959e4b4d8b89e9e7: No bootstrap required, opened a new log
W20260924 03:05:11.750756 13967 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:05:11.750802 13967 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:05:11.752826 13778 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d0c31da3af741dd959e4b4d8b89e9e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0c31da3af741dd959e4b4d8b89e9e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36893 } } peers { permanent_uuid: "484745f9cada4ce0ab1f81bef771eed8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43357 } } peers { permanent_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } }
I20260924 03:05:11.752975 13778 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d0c31da3af741dd959e4b4d8b89e9e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:05:11.750814 13785 master_runner.cc:186] Error getting master registration for 127.0.98.62:36893: OK, instance_id { permanent_uuid: "0d0c31da3af741dd959e4b4d8b89e9e7" instance_seqno: 1790219111290458 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:05:11.753018 13778 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d0c31da3af741dd959e4b4d8b89e9e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d0c31da3af741dd959e4b4d8b89e9e7, State: Initialized, Role: FOLLOWER
I20260924 03:05:11.753072 13785 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:05:11.753123 13778 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d0c31da3af741dd959e4b4d8b89e9e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0c31da3af741dd959e4b4d8b89e9e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36893 } } peers { permanent_uuid: "484745f9cada4ce0ab1f81bef771eed8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43357 } } peers { permanent_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } }
I20260924 03:05:11.753191 13956 raft_consensus.cc:482] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:05:11.753243 13956 raft_consensus.cc:504] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0c31da3af741dd959e4b4d8b89e9e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36893 } } peers { permanent_uuid: "484745f9cada4ce0ab1f81bef771eed8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43357 } } peers { permanent_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } }
I20260924 03:05:11.753269 13934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:05:11.753350 13934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0c31da3af741dd959e4b4d8b89e9e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36893 } } peers { permanent_uuid: "484745f9cada4ce0ab1f81bef771eed8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43357 } } peers { permanent_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } }
I20260924 03:05:11.753396 13778 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d0c31da3af741dd959e4b4d8b89e9e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:05:11.753567 13934 leader_election.cc:288] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d0c31da3af741dd959e4b4d8b89e9e7 (127.0.98.62:36893), 484745f9cada4ce0ab1f81bef771eed8 (127.0.98.61:43357)
I20260924 03:05:11.753763 13902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "484745f9cada4ce0ab1f81bef771eed8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" is_pre_election: true
I20260924 03:05:11.753871 13902 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 484745f9cada4ce0ab1f81bef771eed8 in term 0.
I20260924 03:05:11.753918 13827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "484745f9cada4ce0ab1f81bef771eed8" is_pre_election: true
I20260924 03:05:11.754019 13801 leader_election.cc:302] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171d7dcf3c3a4ab084f5ad999ad003a1, 484745f9cada4ce0ab1f81bef771eed8; no voters:
I20260924 03:05:11.754033 13827 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171d7dcf3c3a4ab084f5ad999ad003a1 in term 0.
I20260924 03:05:11.754290 13877 leader_election.cc:302] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171d7dcf3c3a4ab084f5ad999ad003a1, 484745f9cada4ce0ab1f81bef771eed8; no voters:
I20260924 03:05:11.754391 13934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:05:11.754426 13934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:05:11.754447 13934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:05:11.755271 13934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0c31da3af741dd959e4b4d8b89e9e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36893 } } peers { permanent_uuid: "484745f9cada4ce0ab1f81bef771eed8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43357 } } peers { permanent_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } }
I20260924 03:05:11.755299 13752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d0c31da3af741dd959e4b4d8b89e9e7" is_pre_election: true
I20260924 03:05:11.755398 13934 leader_election.cc:288] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [CANDIDATE]: Term 1 election: Requested vote from peers 0d0c31da3af741dd959e4b4d8b89e9e7 (127.0.98.62:36893), 484745f9cada4ce0ab1f81bef771eed8 (127.0.98.61:43357)
I20260924 03:05:11.755422 13752 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0d0c31da3af741dd959e4b4d8b89e9e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171d7dcf3c3a4ab084f5ad999ad003a1 in term 0.
I20260924 03:05:11.755483 13751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "484745f9cada4ce0ab1f81bef771eed8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d0c31da3af741dd959e4b4d8b89e9e7" is_pre_election: true
I20260924 03:05:11.755645 13751 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0d0c31da3af741dd959e4b4d8b89e9e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 484745f9cada4ce0ab1f81bef771eed8 in term 0.
I20260924 03:05:11.755810 13970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d0c31da3af741dd959e4b4d8b89e9e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0c31da3af741dd959e4b4d8b89e9e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36893 } } peers { permanent_uuid: "484745f9cada4ce0ab1f81bef771eed8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43357 } } peers { permanent_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } } }
I20260924 03:05:11.755960 13970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d0c31da3af741dd959e4b4d8b89e9e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:05:11.755995 13751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d0c31da3af741dd959e4b4d8b89e9e7"
I20260924 03:05:11.756069 13751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d0c31da3af741dd959e4b4d8b89e9e7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:05:11.756879 13751 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0d0c31da3af741dd959e4b4d8b89e9e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171d7dcf3c3a4ab084f5ad999ad003a1 in term 1.
I20260924 03:05:11.757079 13826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "484745f9cada4ce0ab1f81bef771eed8"
I20260924 03:05:11.757158 13826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:05:11.758001 13826 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171d7dcf3c3a4ab084f5ad999ad003a1 in term 1.
I20260924 03:05:11.758419 13876 leader_election.cc:302] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d0c31da3af741dd959e4b4d8b89e9e7, 171d7dcf3c3a4ab084f5ad999ad003a1; no voters:
I20260924 03:05:11.758646 13934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:05:11.758778 13934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [term 1 LEADER]: Becoming Leader. State: Replica: 171d7dcf3c3a4ab084f5ad999ad003a1, State: Running, Role: LEADER
I20260924 03:05:11.758862 13934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0c31da3af741dd959e4b4d8b89e9e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36893 } } peers { permanent_uuid: "484745f9cada4ce0ab1f81bef771eed8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43357 } } peers { permanent_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } }
I20260924 03:05:11.759208 13934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 171d7dcf3c3a4ab084f5ad999ad003a1. Latest consensus state: current_term: 1 leader_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0c31da3af741dd959e4b4d8b89e9e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36893 } } peers { permanent_uuid: "484745f9cada4ce0ab1f81bef771eed8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43357 } } peers { permanent_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } } }
I20260924 03:05:11.759284 13934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [sys.catalog]: This master's current role is: LEADER
I20260924 03:05:11.759420 13956 leader_election.cc:288] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d0c31da3af741dd959e4b4d8b89e9e7 (127.0.98.62:36893), 171d7dcf3c3a4ab084f5ad999ad003a1 (127.0.98.60:38519)
I20260924 03:05:11.759512 13956 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260924 03:05:11.759670 13985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:05:11.759832 13985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:05:11.761204 13827 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8 [term 1 FOLLOWER]: Refusing update from remote peer 171d7dcf3c3a4ab084f5ad999ad003a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:05:11.761423 13751 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d0c31da3af741dd959e4b4d8b89e9e7 [term 1 FOLLOWER]: Refusing update from remote peer 171d7dcf3c3a4ab084f5ad999ad003a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:05:11.761713 13934 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d0c31da3af741dd959e4b4d8b89e9e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:05:11.761821 13983 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "484745f9cada4ce0ab1f81bef771eed8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:05:11.762965 13970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d0c31da3af741dd959e4b4d8b89e9e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 171d7dcf3c3a4ab084f5ad999ad003a1. Latest consensus state: current_term: 1 leader_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0c31da3af741dd959e4b4d8b89e9e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36893 } } peers { permanent_uuid: "484745f9cada4ce0ab1f81bef771eed8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43357 } } peers { permanent_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } } }
I20260924 03:05:11.763051 13970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d0c31da3af741dd959e4b4d8b89e9e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:05:11.764005 13977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 171d7dcf3c3a4ab084f5ad999ad003a1. Latest consensus state: current_term: 1 leader_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0c31da3af741dd959e4b4d8b89e9e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36893 } } peers { permanent_uuid: "484745f9cada4ce0ab1f81bef771eed8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43357 } } peers { permanent_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } } }
I20260924 03:05:11.764127 13977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 03:05:11.765168 13982 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d0c31da3af741dd959e4b4d8b89e9e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:05:11.765264 13982 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:05:11.768292 13986 mvcc.cc:205] Tried to move back new op lower bound from 7332737481772081152 to 7332737481764839424. Current Snapshot: MvccSnapshot[applied={T|T < 7332737481772081152}]
I20260924 03:05:11.768374 13983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0c31da3af741dd959e4b4d8b89e9e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36893 } } peers { permanent_uuid: "484745f9cada4ce0ab1f81bef771eed8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43357 } } peers { permanent_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } } }
I20260924 03:05:11.768466 13983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [sys.catalog]: This master's current role is: LEADER
I20260924 03:05:11.768754 13970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d0c31da3af741dd959e4b4d8b89e9e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0c31da3af741dd959e4b4d8b89e9e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36893 } } peers { permanent_uuid: "484745f9cada4ce0ab1f81bef771eed8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43357 } } peers { permanent_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } } }
I20260924 03:05:11.768826 13970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d0c31da3af741dd959e4b4d8b89e9e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:05:11.769100 13988 mvcc.cc:205] Tried to move back new op lower bound from 7332737481772081152 to 7332737481764839424. Current Snapshot: MvccSnapshot[applied={T|T < 7332737481772081152}]
I20260924 03:05:11.769173 13934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0c31da3af741dd959e4b4d8b89e9e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36893 } } peers { permanent_uuid: "484745f9cada4ce0ab1f81bef771eed8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43357 } } peers { permanent_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } } }
I20260924 03:05:11.769269 13934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1 [sys.catalog]: This master's current role is: LEADER
I20260924 03:05:11.769542 13977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0c31da3af741dd959e4b4d8b89e9e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36893 } } peers { permanent_uuid: "484745f9cada4ce0ab1f81bef771eed8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43357 } } peers { permanent_uuid: "171d7dcf3c3a4ab084f5ad999ad003a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38519 } } }
I20260924 03:05:11.769629 13977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:05:11.774828 13985 catalog_manager.cc:1347] Generated new cluster ID: f53cc9b1f7e74e57baa1efaa3bc04e83
I20260924 03:05:11.774974 13985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:05:11.781288 13985 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:05:11.782001 13985 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:05:11.794108 13985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 171d7dcf3c3a4ab084f5ad999ad003a1: Generated new TSK 0
I20260924 03:05:11.794309 13985 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:05:12.751291 13967 catalog_manager.cc:1259] Loaded cluster ID: f53cc9b1f7e74e57baa1efaa3bc04e83
I20260924 03:05:12.751397 13967 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8: loading cluster ID for follower catalog manager: success
I20260924 03:05:12.752216 13967 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8: acquiring CA information for follower catalog manager: success
I20260924 03:05:12.752525 13967 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 484745f9cada4ce0ab1f81bef771eed8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:05:12.765749 13982 catalog_manager.cc:1259] Loaded cluster ID: f53cc9b1f7e74e57baa1efaa3bc04e83
I20260924 03:05:12.765829 13982 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0d0c31da3af741dd959e4b4d8b89e9e7: loading cluster ID for follower catalog manager: success
I20260924 03:05:12.767172 13982 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0d0c31da3af741dd959e4b4d8b89e9e7: acquiring CA information for follower catalog manager: success
I20260924 03:05:12.767628 13982 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0d0c31da3af741dd959e4b4d8b89e9e7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:05:12.848690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 13710
I20260924 03:05:12.854288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 13785
I20260924 03:05:12.860170 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 13860
2026-09-24T03:05:12Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2907 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:05:12 dist-test-slave-wgsb krb5kdc[14002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:05:12 dist-test-slave-wgsb krb5kdc[14002](info): set up 2 sockets
Sep 24 03:05:12 dist-test-slave-wgsb krb5kdc[14002](info): commencing operation
krb5kdc: starting...
W20260924 03:05:13.907805 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:05:13 dist-test-slave-wgsb krb5kdc[14002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:05:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:05:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:05:14.031976 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40031
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:05:14.117728 14015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:14.118034 14015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:14.118062 14015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:14.119275 14015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:05:14.119323 14015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:14.119338 14015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:14.119350 14015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:05:14.119436 14015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:05:14.120967 14015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:14.121240 14015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:14.121495 14015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:14.126767 14020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:14.126760 14023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:14.126801 14021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:14.127190 14015 server_base.cc:1034] running on GCE node
I20260924 03:05:14.127465 14015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:14.127797 14015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:14.129007 14015 hybrid_clock.cc:648] HybridClock initialized: now 1790219114128985 us; error 41 us; skew 500 ppm
Sep 24 03:05:14 dist-test-slave-wgsb krb5kdc[14002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:14.131548 14015 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:05:14.132061 14015 webserver.cc:460] Webserver started at http://127.0.98.62:39271/ using document root <none> and password file <none>
I20260924 03:05:14.132301 14015 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:14.132370 14015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:14.132524 14015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:05:14.133688 14015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "ef8136f7bdf44731a7b0799769fc21d9"
format_stamp: "Formatted at 2026-09-24 03:05:14 on dist-test-slave-wgsb"
server_key: "01d6b9300d1155e82aa4b64aabd1eeb3"
server_key_iv: "ee21d8de770642340c36e84a7632338a"
server_key_version: "encryptionkey@0"
I20260924 03:05:14.134088 14015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "ef8136f7bdf44731a7b0799769fc21d9"
format_stamp: "Formatted at 2026-09-24 03:05:14 on dist-test-slave-wgsb"
server_key: "01d6b9300d1155e82aa4b64aabd1eeb3"
server_key_iv: "ee21d8de770642340c36e84a7632338a"
server_key_version: "encryptionkey@0"
I20260924 03:05:14.135506 14015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 03:05:14.136340 14030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:14.136515 14015 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:05:14.136622 14015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "ef8136f7bdf44731a7b0799769fc21d9"
format_stamp: "Formatted at 2026-09-24 03:05:14 on dist-test-slave-wgsb"
server_key: "01d6b9300d1155e82aa4b64aabd1eeb3"
server_key_iv: "ee21d8de770642340c36e84a7632338a"
server_key_version: "encryptionkey@0"
I20260924 03:05:14.136713 14015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:14.144908 14015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:14.146984 14015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:14.147161 14015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:14.151352 14015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40031
I20260924 03:05:14.151393 14082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40031 every 8 connection(s)
I20260924 03:05:14.151829 14015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:05:14.152791 14083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:05:14.155290 14083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef8136f7bdf44731a7b0799769fc21d9: Bootstrap starting.
I20260924 03:05:14.156003 14083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef8136f7bdf44731a7b0799769fc21d9: Neither blocks nor log segments found. Creating new log.
I20260924 03:05:14.156333 14083 log.cc:824] T 00000000000000000000000000000000 P ef8136f7bdf44731a7b0799769fc21d9: Log is configured to *not* fsync() on all Append() calls
I20260924 03:05:14.157125 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 14015
I20260924 03:05:14.157225 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:05:14.157231 14083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef8136f7bdf44731a7b0799769fc21d9: No bootstrap required, opened a new log
I20260924 03:05:14.157423 392 external_mini_cluster.cc:1428] Setting key 2bfc931a273b7fc2008e9c6081fbc499
I20260924 03:05:14.159117 14083 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef8136f7bdf44731a7b0799769fc21d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8136f7bdf44731a7b0799769fc21d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40031 } }
I20260924 03:05:14.159291 14083 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef8136f7bdf44731a7b0799769fc21d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:05:14.159382 14083 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef8136f7bdf44731a7b0799769fc21d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef8136f7bdf44731a7b0799769fc21d9, State: Initialized, Role: FOLLOWER
I20260924 03:05:14.159538 14083 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef8136f7bdf44731a7b0799769fc21d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8136f7bdf44731a7b0799769fc21d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40031 } }
I20260924 03:05:14.159667 14083 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef8136f7bdf44731a7b0799769fc21d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:05:14.159711 14083 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef8136f7bdf44731a7b0799769fc21d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:05:14.159765 14083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef8136f7bdf44731a7b0799769fc21d9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:05:14.160768 14083 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef8136f7bdf44731a7b0799769fc21d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8136f7bdf44731a7b0799769fc21d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40031 } }
I20260924 03:05:14.160856 14083 leader_election.cc:302] T 00000000000000000000000000000000 P ef8136f7bdf44731a7b0799769fc21d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef8136f7bdf44731a7b0799769fc21d9; no voters:
I20260924 03:05:14.161078 14083 leader_election.cc:288] T 00000000000000000000000000000000 P ef8136f7bdf44731a7b0799769fc21d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:05:14.161092 14088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef8136f7bdf44731a7b0799769fc21d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:05:14.161219 14088 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef8136f7bdf44731a7b0799769fc21d9 [term 1 LEADER]: Becoming Leader. State: Replica: ef8136f7bdf44731a7b0799769fc21d9, State: Running, Role: LEADER
I20260924 03:05:14.161355 14083 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef8136f7bdf44731a7b0799769fc21d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:05:14.161352 14088 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef8136f7bdf44731a7b0799769fc21d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8136f7bdf44731a7b0799769fc21d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40031 } }
I20260924 03:05:14.161821 14090 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef8136f7bdf44731a7b0799769fc21d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef8136f7bdf44731a7b0799769fc21d9. Latest consensus state: current_term: 1 leader_uuid: "ef8136f7bdf44731a7b0799769fc21d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8136f7bdf44731a7b0799769fc21d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40031 } } }
I20260924 03:05:14.161981 14089 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef8136f7bdf44731a7b0799769fc21d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef8136f7bdf44731a7b0799769fc21d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8136f7bdf44731a7b0799769fc21d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40031 } } }
I20260924 03:05:14.162083 14090 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef8136f7bdf44731a7b0799769fc21d9 [sys.catalog]: This master's current role is: LEADER
I20260924 03:05:14.162340 14089 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef8136f7bdf44731a7b0799769fc21d9 [sys.catalog]: This master's current role is: LEADER
I20260924 03:05:14.162709 14101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:05:14.163270 14101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:05:14.165035 14101 catalog_manager.cc:1347] Generated new cluster ID: a0aee4de4038465b8ba5f320526fabe9
I20260924 03:05:14.165086 14101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:05:14 dist-test-slave-wgsb krb5kdc[14002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:05:14.169098 14087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:14.157970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45270 (local address 127.0.98.62:40031)
0924 03:05:14.158134 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:14.158143 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:14.158184 (+ 41us) server_negotiation.cc:407] Connection header received
0924 03:05:14.159099 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:14.159106 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:14.159163 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:14.159313 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:14.160136 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:14.161144 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:14.162259 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:14.162871 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:14.162986 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:14.165752 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:14.165756 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:14.165757 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:14.165768 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:14.167437 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:14.167875 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:14.167878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:14.167879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:14.167991 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:14.168207 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:14.168208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:14.168208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:14.168460 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:14.168631 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:14.168709 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 03:05:14.168952 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260924 03:05:14.169955 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:05:14.185493 14101 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:05:14.186074 14101 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:05:14.188138 14101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef8136f7bdf44731a7b0799769fc21d9: Generated new TSK 0
I20260924 03:05:14.188324 14101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:05:14.223948 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 14015
2026-09-24T03:05:14Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1362 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:05:14 dist-test-slave-wgsb krb5kdc[14110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:05:14 dist-test-slave-wgsb krb5kdc[14110](info): set up 2 sockets
Sep 24 03:05:14 dist-test-slave-wgsb krb5kdc[14110](info): commencing operation
krb5kdc: starting...
W20260924 03:05:15.319753 392 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:05:15 dist-test-slave-wgsb krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:05:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:05:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:05:15.451931 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39953
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:05:15.533286 14123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:15.533529 14123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:15.533566 14123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:15.534711 14123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:05:15.534775 14123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:15.534801 14123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:15.534827 14123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:05:15.534858 14123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:05:15.536685 14123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:15.536967 14123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:15.537199 14123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:15.542121 14129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:15.542078 14128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:15.542083 14131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:15.542346 14123 server_base.cc:1034] running on GCE node
I20260924 03:05:15.542718 14123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:15.543017 14123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:15.544157 14123 hybrid_clock.cc:648] HybridClock initialized: now 1790219115544144 us; error 32 us; skew 500 ppm
Sep 24 03:05:15 dist-test-slave-wgsb krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:15.546766 14123 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:05:15.547277 14123 webserver.cc:460] Webserver started at http://127.0.98.62:34447/ using document root <none> and password file <none>
I20260924 03:05:15.547514 14123 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:15.547582 14123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:15.547744 14123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:05:15.548804 14123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "c5beaf31902c40e2baea7c9ea29b4712"
format_stamp: "Formatted at 2026-09-24 03:05:15 on dist-test-slave-wgsb"
server_key: "023839d15d37656f4b66540ccfe905fb"
server_key_iv: "88f7278b939ee3e63de490bc3a4baf9d"
server_key_version: "encryptionkey@0"
I20260924 03:05:15.549180 14123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "c5beaf31902c40e2baea7c9ea29b4712"
format_stamp: "Formatted at 2026-09-24 03:05:15 on dist-test-slave-wgsb"
server_key: "023839d15d37656f4b66540ccfe905fb"
server_key_iv: "88f7278b939ee3e63de490bc3a4baf9d"
server_key_version: "encryptionkey@0"
I20260924 03:05:15.550575 14123 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:05:15.551362 14138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:15.551563 14123 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:05:15.551685 14123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "c5beaf31902c40e2baea7c9ea29b4712"
format_stamp: "Formatted at 2026-09-24 03:05:15 on dist-test-slave-wgsb"
server_key: "023839d15d37656f4b66540ccfe905fb"
server_key_iv: "88f7278b939ee3e63de490bc3a4baf9d"
server_key_version: "encryptionkey@0"
I20260924 03:05:15.551757 14123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:15.564301 14123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:15.566390 14123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:15.566565 14123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:15.570905 14123 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39953
I20260924 03:05:15.570988 14190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39953 every 8 connection(s)
I20260924 03:05:15.571317 14123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:05:15.572420 14191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:05:15.574846 14191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712: Bootstrap starting.
I20260924 03:05:15.575448 14191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712: Neither blocks nor log segments found. Creating new log.
I20260924 03:05:15.575716 14191 log.cc:824] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712: Log is configured to *not* fsync() on all Append() calls
I20260924 03:05:15.576390 14191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712: No bootstrap required, opened a new log
I20260924 03:05:15.577172 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 14123
I20260924 03:05:15.577302 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:05:15.577515 392 external_mini_cluster.cc:1428] Setting key 281213fb771d4f45614c7e26e5c32fd1
I20260924 03:05:15.578612 14191 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5beaf31902c40e2baea7c9ea29b4712" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39953 } }
I20260924 03:05:15.578795 14191 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:05:15.578864 14191 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5beaf31902c40e2baea7c9ea29b4712, State: Initialized, Role: FOLLOWER
I20260924 03:05:15.579038 14191 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5beaf31902c40e2baea7c9ea29b4712" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39953 } }
I20260924 03:05:15.579147 14191 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:05:15.579197 14191 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:05:15.579254 14191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:05:15.580300 14191 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5beaf31902c40e2baea7c9ea29b4712" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39953 } }
I20260924 03:05:15.580437 14191 leader_election.cc:302] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5beaf31902c40e2baea7c9ea29b4712; no voters:
I20260924 03:05:15.580654 14191 leader_election.cc:288] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:05:15.580679 14196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:05:15.580863 14196 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [term 1 LEADER]: Becoming Leader. State: Replica: c5beaf31902c40e2baea7c9ea29b4712, State: Running, Role: LEADER
I20260924 03:05:15.581024 14191 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:05:15.581009 14196 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5beaf31902c40e2baea7c9ea29b4712" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39953 } }
W20260924 03:05:15.583030 14211 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:05:15.583099 14211 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:05:15.581684 14198 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5beaf31902c40e2baea7c9ea29b4712" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5beaf31902c40e2baea7c9ea29b4712" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39953 } } }
I20260924 03:05:15.581741 14197 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5beaf31902c40e2baea7c9ea29b4712. Latest consensus state: current_term: 1 leader_uuid: "c5beaf31902c40e2baea7c9ea29b4712" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5beaf31902c40e2baea7c9ea29b4712" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39953 } } }
I20260924 03:05:15.583199 14198 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [sys.catalog]: This master's current role is: LEADER
I20260924 03:05:15.583217 14197 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [sys.catalog]: This master's current role is: LEADER
I20260924 03:05:15.583518 14213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:05:15.583747 14213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 03:05:15 dist-test-slave-wgsb krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:05:15.585613 14213 catalog_manager.cc:1347] Generated new cluster ID: e0338a1bcc984f589a4ca3b9a163155d
I20260924 03:05:15.585667 14213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:05:15.588879 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:15.578130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35912 (local address 127.0.98.62:39953)
0924 03:05:15.578297 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:15.578306 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:15.578335 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:05:15.578615 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:15.578624 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:15.578682 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:15.578859 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:15.579657 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:15.580651 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:15.581761 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:15.582392 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:15.582493 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:15.585418 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:15.585422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:15.585422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:15.585435 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:15.587243 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:15.587649 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:15.587652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:15.587653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:15.587769 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:15.588004 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:15.588005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:15.588005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:15.588255 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:15.588442 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:15.588521 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 03:05:15.588668 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260924 03:05:15.618530 14213 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:05:15.619262 14213 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:05:15.631276 14213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712: Generated new TSK 0
I20260924 03:05:15.631511 14213 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:05:15.672951 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39953
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:05:15.765292 14217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:15.765511 14217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:15.765537 14217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:15.766659 14217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:15.766716 14217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:15.766744 14217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:05:15.768378 14217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:15.768579 14217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:15.768759 14217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:15.774308 14222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:15.774330 14225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:15.774344 14223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:15.774677 14217 server_base.cc:1034] running on GCE node
I20260924 03:05:15.774883 14217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:15.775157 14217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:15.776348 14217 hybrid_clock.cc:648] HybridClock initialized: now 1790219115776336 us; error 29 us; skew 500 ppm
Sep 24 03:05:15 dist-test-slave-wgsb krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:15.779178 14217 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:05:15.779661 14217 webserver.cc:460] Webserver started at http://127.0.98.1:33163/ using document root <none> and password file <none>
I20260924 03:05:15.779888 14217 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:15.779934 14217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:15.780042 14217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:05:15.781186 14217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/data/instance:
uuid: "47b995a4829f4aadbf44f7f523331774"
format_stamp: "Formatted at 2026-09-24 03:05:15 on dist-test-slave-wgsb"
server_key: "d0bb61df205658d4d361c4860cfe0048"
server_key_iv: "783c6e5b5382b3ec81e072acc1412f7a"
server_key_version: "encryptionkey@0"
I20260924 03:05:15.781524 14217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/wal/instance:
uuid: "47b995a4829f4aadbf44f7f523331774"
format_stamp: "Formatted at 2026-09-24 03:05:15 on dist-test-slave-wgsb"
server_key: "d0bb61df205658d4d361c4860cfe0048"
server_key_iv: "783c6e5b5382b3ec81e072acc1412f7a"
server_key_version: "encryptionkey@0"
I20260924 03:05:15.783121 14217 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:05:15.783980 14232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:15.784138 14217 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:05:15.784219 14217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "47b995a4829f4aadbf44f7f523331774"
format_stamp: "Formatted at 2026-09-24 03:05:15 on dist-test-slave-wgsb"
server_key: "d0bb61df205658d4d361c4860cfe0048"
server_key_iv: "783c6e5b5382b3ec81e072acc1412f7a"
server_key_version: "encryptionkey@0"
I20260924 03:05:15.784274 14217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:15.796720 14217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:15.798556 14217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:15.798666 14217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:15.798846 14217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:05:15.799104 14217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:05:15.799134 14217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:15.799156 14217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:05:15.799168 14217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:15.804950 14217 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41695
I20260924 03:05:15.805081 14345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41695 every 8 connection(s)
I20260924 03:05:15.805359 14217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:05:15.808773 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 14217
I20260924 03:05:15.808881 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:05:15.809064 392 external_mini_cluster.cc:1428] Setting key fa914bf50a7c72fef94beeac26d42a62
Sep 24 03:05:15 dist-test-slave-wgsb krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:05:15.815560 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:15.806088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54855 (local address 127.0.98.62:39953)
0924 03:05:15.806154 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:15.806156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:15.806342 (+ 186us) server_negotiation.cc:407] Connection header received
0924 03:05:15.806863 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:15.806865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:15.806881 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:15.806929 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:15.807972 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:15.808490 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:15.809857 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:15.810077 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:15.810137 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:15.812717 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:15.812719 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:15.812719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:15.812727 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:15.814142 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:15.814573 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:15.814575 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:15.814575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:15.814634 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:15.814888 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:15.814890 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:15.814890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:15.815012 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:15.815083 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:15.815368 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 03:05:15.815468 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 03:05:15.815733 14348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:15.806069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.1:54855)
0924 03:05:15.806221 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:15.806261 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:15.806759 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:15.807008 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:15.807011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:15.807578 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:15.807834 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:15.807838 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:15.808589 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:15.808590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:15.809777 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:15.809780 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:15.810166 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:15.810167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:15.810325 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:15.810824 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:15.810835 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:15.812616 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:15.814243 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:15.814250 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:15.814253 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:15.814482 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:15.814701 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:15.814702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:15.814703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:15.814813 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:15.815089 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:15.815096 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:15.815230 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 03:05:15.815329 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 03:05:15.815487 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260924 03:05:15.816395 14346 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39953
I20260924 03:05:15.816501 14346 heartbeater.cc:464] Registering TS with master...
I20260924 03:05:15.816716 14346 heartbeater.cc:511] Master 127.0.98.62:39953 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 03:05:15.817304 14155 ts_manager.cc:194] Registered new tserver with Master: 47b995a4829f4aadbf44f7f523331774 (127.0.98.1:41695)
I20260924 03:05:15.817981 14155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:05:15.836899 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39953
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:05:15.919368 14351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:15.919613 14351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:15.919644 14351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:15.920774 14351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:15.920814 14351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:15.920840 14351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:05:15.922236 14351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:15.922435 14351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:15.922621 14351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:15.928401 14359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:15.928378 14356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:15.928429 14351 server_base.cc:1034] running on GCE node
W20260924 03:05:15.928392 14357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:15.928776 14351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:15.929047 14351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:15.930238 14351 hybrid_clock.cc:648] HybridClock initialized: now 1790219115930217 us; error 39 us; skew 500 ppm
Sep 24 03:05:15 dist-test-slave-wgsb krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:15.932898 14351 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:05:15.933393 14351 webserver.cc:460] Webserver started at http://127.0.98.2:44969/ using document root <none> and password file <none>
I20260924 03:05:15.933636 14351 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:15.933705 14351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:15.933861 14351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:05:15.934901 14351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/data/instance:
uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502"
format_stamp: "Formatted at 2026-09-24 03:05:15 on dist-test-slave-wgsb"
server_key: "1d582927be7b38cf73874e300f1c9c5c"
server_key_iv: "dfa7fe97d9231262bcaf45a129a391ae"
server_key_version: "encryptionkey@0"
I20260924 03:05:15.935261 14351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502"
format_stamp: "Formatted at 2026-09-24 03:05:15 on dist-test-slave-wgsb"
server_key: "1d582927be7b38cf73874e300f1c9c5c"
server_key_iv: "dfa7fe97d9231262bcaf45a129a391ae"
server_key_version: "encryptionkey@0"
I20260924 03:05:15.936662 14351 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:05:15.937515 14366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:15.937685 14351 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:05:15.937852 14351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502"
format_stamp: "Formatted at 2026-09-24 03:05:15 on dist-test-slave-wgsb"
server_key: "1d582927be7b38cf73874e300f1c9c5c"
server_key_iv: "dfa7fe97d9231262bcaf45a129a391ae"
server_key_version: "encryptionkey@0"
I20260924 03:05:15.937945 14351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:15.950505 14351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:15.952592 14351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:15.952776 14351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:15.953042 14351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:05:15.953392 14351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:05:15.953459 14351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:15.953505 14351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:05:15.953536 14351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:15.959669 14351 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34387
I20260924 03:05:15.959676 14479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34387 every 8 connection(s)
I20260924 03:05:15.960072 14351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:05:15.961865 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 14351
I20260924 03:05:15.962000 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:05:15.962177 392 external_mini_cluster.cc:1428] Setting key 3772030d945112e559ad641a2536b676
Sep 24 03:05:15 dist-test-slave-wgsb krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:05:15.970903 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:15.960784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55685 (local address 127.0.98.62:39953)
0924 03:05:15.960830 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:15.960832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:15.961033 (+ 201us) server_negotiation.cc:407] Connection header received
0924 03:05:15.961580 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:15.961581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:15.961595 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:15.961652 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:15.962874 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:15.963369 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:15.964766 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:15.965088 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:15.965168 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:15.967865 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:15.967866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:15.967867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:15.967875 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:15.969488 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:15.969883 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:15.969885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:15.969885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:15.969948 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:15.970191 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:15.970192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:15.970192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:15.970358 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:15.970447 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:15.970712 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 03:05:15.970808 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:05:15.970989 14482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:15.960758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.2:55685)
0924 03:05:15.960895 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:15.960941 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:15.961468 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:15.961739 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:15.961743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:15.962498 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:15.962742 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:15.962748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:15.963492 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:15.963493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:15.964671 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:15.964674 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:15.965175 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:15.965175 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:15.965272 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:15.965905 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:15.965915 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:15.967771 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:15.969576 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:15.969582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:15.969583 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:15.969805 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:15.970025 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:15.970026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:15.970026 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:15.970124 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:15.970456 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:15.970460 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:15.970577 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 03:05:15.970664 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 03:05:15.970788 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 03:05:15.971789 14480 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39953
I20260924 03:05:15.971956 14480 heartbeater.cc:464] Registering TS with master...
I20260924 03:05:15.972169 14480 heartbeater.cc:511] Master 127.0.98.62:39953 requested a full tablet report, sending...
I20260924 03:05:15.972654 14155 ts_manager.cc:194] Registered new tserver with Master: 3d5e3a3c37cd4aeb8c39f300acc4d502 (127.0.98.2:34387)
I20260924 03:05:15.973127 14155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:05:15.991493 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39953
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:05:16.076332 14485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:16.076558 14485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:16.076584 14485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:16.077891 14485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:16.077940 14485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:16.077970 14485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:05:16.079583 14485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:16.079886 14485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:16.080148 14485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:16.086755 14493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:16.086766 14490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:16.086846 14485 server_base.cc:1034] running on GCE node
W20260924 03:05:16.086763 14491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:16.087309 14485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:16.087634 14485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:16.088824 14485 hybrid_clock.cc:648] HybridClock initialized: now 1790219116088807 us; error 40 us; skew 500 ppm
Sep 24 03:05:16 dist-test-slave-wgsb krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:16.091566 14485 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:05:16.092068 14485 webserver.cc:460] Webserver started at http://127.0.98.3:46089/ using document root <none> and password file <none>
I20260924 03:05:16.092314 14485 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:16.092361 14485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:16.092494 14485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:05:16.093546 14485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/data/instance:
uuid: "be77a11fa86949e0a5c7cdf9af71f901"
format_stamp: "Formatted at 2026-09-24 03:05:16 on dist-test-slave-wgsb"
server_key: "9721af1ac5b03b5969da67ae425ee356"
server_key_iv: "74f650090565e0613c1136ac3c1bc00f"
server_key_version: "encryptionkey@0"
I20260924 03:05:16.093894 14485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/wal/instance:
uuid: "be77a11fa86949e0a5c7cdf9af71f901"
format_stamp: "Formatted at 2026-09-24 03:05:16 on dist-test-slave-wgsb"
server_key: "9721af1ac5b03b5969da67ae425ee356"
server_key_iv: "74f650090565e0613c1136ac3c1bc00f"
server_key_version: "encryptionkey@0"
I20260924 03:05:16.095219 14485 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:05:16.096037 14500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:16.096192 14485 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 03:05:16.096282 14485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "be77a11fa86949e0a5c7cdf9af71f901"
format_stamp: "Formatted at 2026-09-24 03:05:16 on dist-test-slave-wgsb"
server_key: "9721af1ac5b03b5969da67ae425ee356"
server_key_iv: "74f650090565e0613c1136ac3c1bc00f"
server_key_version: "encryptionkey@0"
I20260924 03:05:16.096366 14485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:16.106566 14485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:16.108510 14485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:16.108632 14485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:16.109447 14485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:05:16.109948 14485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:05:16.109989 14485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:16.110013 14485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:05:16.110059 14485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:16.115924 14485 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35215
I20260924 03:05:16.116024 14613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35215 every 8 connection(s)
I20260924 03:05:16.116320 14485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:05:16.116762 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 14485
I20260924 03:05:16.116897 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:05:16.117096 392 external_mini_cluster.cc:1428] Setting key bd0b8530ef9a117343f04d846874c97c
Sep 24 03:05:16 dist-test-slave-wgsb krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:05:16.126246 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.117096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45847 (local address 127.0.98.62:39953)
0924 03:05:16.117153 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:16.117156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:16.117432 (+ 276us) server_negotiation.cc:407] Connection header received
0924 03:05:16.118018 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:16.118019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:16.118031 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:16.118088 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:16.119031 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.119541 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.120571 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.120783 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.120875 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.123199 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:16.123200 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:16.123201 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:16.123211 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:16.124764 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:16.125222 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:16.125223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:16.125224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:16.125283 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:16.125553 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:16.125554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:16.125554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:16.125676 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:16.125750 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:16.126061 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 03:05:16.126154 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 03:05:16.126435 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.117100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.3:45847)
0924 03:05:16.117297 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:16.117353 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:16.117916 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:16.118158 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:16.118162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:16.118732 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:16.118930 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:16.118934 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.119668 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.119669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:16.120474 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:16.120476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.120887 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.120888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:16.121021 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.121555 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:16.121576 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:16.123091 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:16.124869 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:16.124877 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:16.124879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:16.125130 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:16.125352 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:16.125353 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:16.125353 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:16.125472 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:16.125754 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:16.125761 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:16.125905 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 03:05:16.126026 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 03:05:16.126190 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":103,"threads_started":1}
I20260924 03:05:16.127110 14614 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39953
I20260924 03:05:16.127197 14614 heartbeater.cc:464] Registering TS with master...
I20260924 03:05:16.127395 14614 heartbeater.cc:511] Master 127.0.98.62:39953 requested a full tablet report, sending...
I20260924 03:05:16.127890 14155 ts_manager.cc:194] Registered new tserver with Master: be77a11fa86949e0a5c7cdf9af71f901 (127.0.98.3:35215)
I20260924 03:05:16.128269 14155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45847
I20260924 03:05:16.129150 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:05:16.137293 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.130524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35928 (local address 127.0.98.62:39953)
0924 03:05:16.130594 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:16.130596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:16.130692 (+ 96us) server_negotiation.cc:407] Connection header received
0924 03:05:16.130770 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:16.130771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:16.130780 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:16.130825 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:16.131409 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.132029 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.132781 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.132990 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.133075 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.134321 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:16.134323 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:16.134323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:16.134330 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:16.135902 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:16.136334 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:16.136336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:16.136336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:16.136407 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:16.136624 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:16.136625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:16.136626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:16.136798 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:16.136917 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:16.137042 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 03:05:16.137126 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 03:05:16.138886 14155 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:05:16.140765 14155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:05:16.147742 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.143146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35215 (local address 127.0.0.1:38562)
0924 03:05:16.143308 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:16.143353 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:16.143425 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:16.143817 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:16.143820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:16.143822 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:05:16.143903 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:16.143905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.145148 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.145149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:16.146572 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:16.146575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.147383 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.147385 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:16.147472 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.147487 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 03:05:16.147569 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 03:05:16.147657 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":50,"threads_started":1}
I20260924 03:05:16.147755 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.143365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38562 (local address 127.0.98.3:35215)
0924 03:05:16.143615 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:16.143620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:16.143633 (+ 13us) server_negotiation.cc:407] Connection header received
0924 03:05:16.143683 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:16.143686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:16.143705 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:16.143793 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:05:16.143980 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.145029 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.146680 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.147267 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.147379 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.147455 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:16.147617 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 03:05:16.147659 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":87,"threads_started":1}
I20260924 03:05:16.147914 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.143146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34387 (local address 127.0.0.1:33092)
0924 03:05:16.143320 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:16.143376 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:16.143432 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:16.144174 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:16.144175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:16.144177 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:05:16.144221 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:16.144223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.145513 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.145514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:16.146715 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:16.146718 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.147623 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.147624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:16.147696 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.147705 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 03:05:16.147755 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 03:05:16.147810 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":12,"thread_start_us":97,"threads_started":1}
I20260924 03:05:16.148150 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.143575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33092 (local address 127.0.98.2:34387)
0924 03:05:16.143801 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:16.143816 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:16.144035 (+ 219us) server_negotiation.cc:407] Connection header received
0924 03:05:16.144078 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:16.144080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:16.144097 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:16.144178 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:05:16.144303 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.145390 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.146783 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.147455 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.147512 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.147563 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:16.148033 (+ 470us) server_negotiation.cc:299] Negotiation successful
0924 03:05:16.148073 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":124,"threads_started":1}
I20260924 03:05:16.148592 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.143156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41695 (local address 127.0.0.1:57036)
0924 03:05:16.143370 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:16.143379 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:16.143432 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:16.144499 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:16.144500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:16.144503 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:05:16.144541 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:16.144543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.145741 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.145742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:16.146611 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:16.146614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.147641 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.147641 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:16.147696 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.147702 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 03:05:16.147755 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 03:05:16.148490 (+ 735us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":663,"thread_start_us":85,"threads_started":1}
I20260924 03:05:16.148907 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.143885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57036 (local address 127.0.98.1:41695)
0924 03:05:16.144324 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:16.144329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:16.144340 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:05:16.144387 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:16.144389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:16.144404 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:16.144485 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:05:16.144615 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.145647 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.146710 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.147433 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.147509 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.147563 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:16.148774 (+ 1211us) server_negotiation.cc:299] Negotiation successful
0924 03:05:16.148825 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":87,"threads_started":1}
I20260924 03:05:16.148917 14414 tablet_service.cc:1467] Processing CreateTablet for tablet 12dbf408ec95486982e3284a4b7a2740 (DEFAULT_TABLE table=test-table [id=af5d824447984492ba3a58735f29eb30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:05:16.148955 14548 tablet_service.cc:1467] Processing CreateTablet for tablet 12dbf408ec95486982e3284a4b7a2740 (DEFAULT_TABLE table=test-table [id=af5d824447984492ba3a58735f29eb30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:05:16.149328 14414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12dbf408ec95486982e3284a4b7a2740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:05:16.149327 14548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12dbf408ec95486982e3284a4b7a2740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:05:16.149523 14280 tablet_service.cc:1467] Processing CreateTablet for tablet 12dbf408ec95486982e3284a4b7a2740 (DEFAULT_TABLE table=test-table [id=af5d824447984492ba3a58735f29eb30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:05:16.149844 14280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12dbf408ec95486982e3284a4b7a2740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:05:16.152448 14631 tablet_bootstrap.cc:492] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774: Bootstrap starting.
I20260924 03:05:16.152981 14633 tablet_bootstrap.cc:492] T 12dbf408ec95486982e3284a4b7a2740 P be77a11fa86949e0a5c7cdf9af71f901: Bootstrap starting.
I20260924 03:05:16.153271 14631 tablet_bootstrap.cc:654] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774: Neither blocks nor log segments found. Creating new log.
I20260924 03:05:16.154017 14633 tablet_bootstrap.cc:654] T 12dbf408ec95486982e3284a4b7a2740 P be77a11fa86949e0a5c7cdf9af71f901: Neither blocks nor log segments found. Creating new log.
I20260924 03:05:16.154096 14631 log.cc:824] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774: Log is configured to *not* fsync() on all Append() calls
I20260924 03:05:16.154677 14633 log.cc:824] T 12dbf408ec95486982e3284a4b7a2740 P be77a11fa86949e0a5c7cdf9af71f901: Log is configured to *not* fsync() on all Append() calls
I20260924 03:05:16.154932 14631 tablet_bootstrap.cc:492] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774: No bootstrap required, opened a new log
I20260924 03:05:16.154999 14631 ts_tablet_manager.cc:1397] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 03:05:16.155014 14632 tablet_bootstrap.cc:492] T 12dbf408ec95486982e3284a4b7a2740 P 3d5e3a3c37cd4aeb8c39f300acc4d502: Bootstrap starting.
I20260924 03:05:16.155661 14632 tablet_bootstrap.cc:654] T 12dbf408ec95486982e3284a4b7a2740 P 3d5e3a3c37cd4aeb8c39f300acc4d502: Neither blocks nor log segments found. Creating new log.
I20260924 03:05:16.156077 14633 tablet_bootstrap.cc:492] T 12dbf408ec95486982e3284a4b7a2740 P be77a11fa86949e0a5c7cdf9af71f901: No bootstrap required, opened a new log
I20260924 03:05:16.156169 14633 ts_tablet_manager.cc:1397] T 12dbf408ec95486982e3284a4b7a2740 P be77a11fa86949e0a5c7cdf9af71f901: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 03:05:16.156204 14632 log.cc:824] T 12dbf408ec95486982e3284a4b7a2740 P 3d5e3a3c37cd4aeb8c39f300acc4d502: Log is configured to *not* fsync() on all Append() calls
I20260924 03:05:16.156998 14632 tablet_bootstrap.cc:492] T 12dbf408ec95486982e3284a4b7a2740 P 3d5e3a3c37cd4aeb8c39f300acc4d502: No bootstrap required, opened a new log
I20260924 03:05:16.157068 14632 ts_tablet_manager.cc:1397] T 12dbf408ec95486982e3284a4b7a2740 P 3d5e3a3c37cd4aeb8c39f300acc4d502: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 03:05:16.157217 14631 raft_consensus.cc:348] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } }
I20260924 03:05:16.157405 14631 raft_consensus.cc:374] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:05:16.157442 14631 raft_consensus.cc:729] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47b995a4829f4aadbf44f7f523331774, State: Initialized, Role: FOLLOWER
I20260924 03:05:16.157582 14631 consensus_queue.cc:260] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } }
I20260924 03:05:16.157857 14631 ts_tablet_manager.cc:1428] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:05:16.157982 14346 heartbeater.cc:503] Master 127.0.98.62:39953 was elected leader, sending a full tablet report...
I20260924 03:05:16.158241 14633 raft_consensus.cc:348] T 12dbf408ec95486982e3284a4b7a2740 P be77a11fa86949e0a5c7cdf9af71f901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } }
I20260924 03:05:16.158432 14633 raft_consensus.cc:374] T 12dbf408ec95486982e3284a4b7a2740 P be77a11fa86949e0a5c7cdf9af71f901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:05:16.158483 14633 raft_consensus.cc:729] T 12dbf408ec95486982e3284a4b7a2740 P be77a11fa86949e0a5c7cdf9af71f901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be77a11fa86949e0a5c7cdf9af71f901, State: Initialized, Role: FOLLOWER
I20260924 03:05:16.158599 14633 consensus_queue.cc:260] T 12dbf408ec95486982e3284a4b7a2740 P be77a11fa86949e0a5c7cdf9af71f901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } }
I20260924 03:05:16.158887 14614 heartbeater.cc:503] Master 127.0.98.62:39953 was elected leader, sending a full tablet report...
I20260924 03:05:16.158890 14633 ts_tablet_manager.cc:1428] T 12dbf408ec95486982e3284a4b7a2740 P be77a11fa86949e0a5c7cdf9af71f901: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:05:16.159480 14632 raft_consensus.cc:348] T 12dbf408ec95486982e3284a4b7a2740 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } }
I20260924 03:05:16.159684 14632 raft_consensus.cc:374] T 12dbf408ec95486982e3284a4b7a2740 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:05:16.159734 14632 raft_consensus.cc:729] T 12dbf408ec95486982e3284a4b7a2740 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d5e3a3c37cd4aeb8c39f300acc4d502, State: Initialized, Role: FOLLOWER
I20260924 03:05:16.159837 14632 consensus_queue.cc:260] T 12dbf408ec95486982e3284a4b7a2740 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } }
I20260924 03:05:16.160184 14480 heartbeater.cc:503] Master 127.0.98.62:39953 was elected leader, sending a full tablet report...
I20260924 03:05:16.160122 14632 ts_tablet_manager.cc:1428] T 12dbf408ec95486982e3284a4b7a2740 P 3d5e3a3c37cd4aeb8c39f300acc4d502: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260924 03:05:16.210656 14481 tablet.cc:2369] T 12dbf408ec95486982e3284a4b7a2740 P 3d5e3a3c37cd4aeb8c39f300acc4d502: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:05:16.228237 14637 raft_consensus.cc:482] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:05:16.228392 14637 raft_consensus.cc:504] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } }
I20260924 03:05:16.228873 14637 leader_election.cc:288] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be77a11fa86949e0a5c7cdf9af71f901 (127.0.98.3:35215), 3d5e3a3c37cd4aeb8c39f300acc4d502 (127.0.98.2:34387)
I20260924 03:05:16.232910 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.229088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34387 (local address 127.0.98.1:47635)
0924 03:05:16.229381 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:16.229390 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:16.229443 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:16.229576 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:16.229577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:16.229578 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:05:16.229679 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:16.229680 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.230519 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.230520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:16.231786 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:16.231790 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.232620 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.232622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:16.232703 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.232707 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:05:16.232773 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 03:05:16.232816 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":21,"thread_start_us":124,"threads_started":1}
I20260924 03:05:16.232971 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.229153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47635 (local address 127.0.98.2:34387)
0924 03:05:16.229242 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:16.229244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:16.229454 (+ 210us) server_negotiation.cc:407] Connection header received
0924 03:05:16.229489 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:16.229490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:16.229503 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:16.229557 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:05:16.229758 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.230465 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.231970 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.232525 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.232594 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.232604 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:16.232848 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 03:05:16.232902 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 03:05:16.233016 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.229128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59145 (local address 127.0.98.3:35215)
0924 03:05:16.229233 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:16.229235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:16.229247 (+ 12us) server_negotiation.cc:407] Connection header received
0924 03:05:16.229279 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:16.229280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:16.229294 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:16.229382 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:05:16.229565 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.230292 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.231918 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.232510 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.232594 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.232604 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:16.232900 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 03:05:16.232945 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 03:05:16.233017 14348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.229087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35215 (local address 127.0.98.1:59145)
0924 03:05:16.229145 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:16.229159 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:16.229203 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:16.229406 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:16.229407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:16.229409 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:05:16.229486 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:16.229488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.230406 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.230407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:16.231790 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:16.231791 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.232620 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.232622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:16.232707 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.232709 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:05:16.232774 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 03:05:16.232882 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":37}
I20260924 03:05:16.233424 14568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12dbf408ec95486982e3284a4b7a2740" candidate_uuid: "47b995a4829f4aadbf44f7f523331774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be77a11fa86949e0a5c7cdf9af71f901" is_pre_election: true
I20260924 03:05:16.233444 14434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12dbf408ec95486982e3284a4b7a2740" candidate_uuid: "47b995a4829f4aadbf44f7f523331774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" is_pre_election: true
I20260924 03:05:16.233604 14568 raft_consensus.cc:2454] T 12dbf408ec95486982e3284a4b7a2740 P be77a11fa86949e0a5c7cdf9af71f901 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47b995a4829f4aadbf44f7f523331774 in term 0.
I20260924 03:05:16.233604 14434 raft_consensus.cc:2454] T 12dbf408ec95486982e3284a4b7a2740 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47b995a4829f4aadbf44f7f523331774 in term 0.
I20260924 03:05:16.233847 14235 leader_election.cc:302] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47b995a4829f4aadbf44f7f523331774, be77a11fa86949e0a5c7cdf9af71f901; no voters:
I20260924 03:05:16.234066 14637 raft_consensus.cc:2781] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:05:16.234146 14637 raft_consensus.cc:482] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:05:16.234176 14637 raft_consensus.cc:3037] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:05:16.235003 14637 raft_consensus.cc:504] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } }
I20260924 03:05:16.235157 14637 leader_election.cc:288] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774 [CANDIDATE]: Term 1 election: Requested vote from peers be77a11fa86949e0a5c7cdf9af71f901 (127.0.98.3:35215), 3d5e3a3c37cd4aeb8c39f300acc4d502 (127.0.98.2:34387)
I20260924 03:05:16.235337 14568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12dbf408ec95486982e3284a4b7a2740" candidate_uuid: "47b995a4829f4aadbf44f7f523331774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be77a11fa86949e0a5c7cdf9af71f901"
I20260924 03:05:16.235437 14434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12dbf408ec95486982e3284a4b7a2740" candidate_uuid: "47b995a4829f4aadbf44f7f523331774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502"
I20260924 03:05:16.235450 14568 raft_consensus.cc:3037] T 12dbf408ec95486982e3284a4b7a2740 P be77a11fa86949e0a5c7cdf9af71f901 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:05:16.235522 14434 raft_consensus.cc:3037] T 12dbf408ec95486982e3284a4b7a2740 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:05:16.236442 14568 raft_consensus.cc:2454] T 12dbf408ec95486982e3284a4b7a2740 P be77a11fa86949e0a5c7cdf9af71f901 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47b995a4829f4aadbf44f7f523331774 in term 1.
I20260924 03:05:16.236442 14434 raft_consensus.cc:2454] T 12dbf408ec95486982e3284a4b7a2740 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47b995a4829f4aadbf44f7f523331774 in term 1.
I20260924 03:05:16.236872 14236 leader_election.cc:302] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d5e3a3c37cd4aeb8c39f300acc4d502, 47b995a4829f4aadbf44f7f523331774; no voters:
I20260924 03:05:16.237023 14637 raft_consensus.cc:2781] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:05:16.237231 14637 raft_consensus.cc:686] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774 [term 1 LEADER]: Becoming Leader. State: Replica: 47b995a4829f4aadbf44f7f523331774, State: Running, Role: LEADER
I20260924 03:05:16.237315 14637 consensus_queue.cc:237] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } }
I20260924 03:05:16.238243 14155 catalog_manager.cc:5495] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47b995a4829f4aadbf44f7f523331774 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "47b995a4829f4aadbf44f7f523331774" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } health_report { overall_health: HEALTHY } } }
I20260924 03:05:16.244537 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.241840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57050 (local address 127.0.98.1:41695)
0924 03:05:16.241905 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:16.241907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:16.241927 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:05:16.241965 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:16.241966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:16.241984 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:16.242038 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:05:16.242182 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.242695 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.243532 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.243889 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.243959 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.244149 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:05:16.244256 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:05:16.244342 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:16.244364 (+ 22us) server_negotiation.cc:299] Negotiation successful
0924 03:05:16.244388 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 03:05:16.246498 14434 raft_consensus.cc:1261] T 12dbf408ec95486982e3284a4b7a2740 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [term 1 FOLLOWER]: Refusing update from remote peer 47b995a4829f4aadbf44f7f523331774: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:05:16.246498 14568 raft_consensus.cc:1261] T 12dbf408ec95486982e3284a4b7a2740 P be77a11fa86949e0a5c7cdf9af71f901 [term 1 FOLLOWER]: Refusing update from remote peer 47b995a4829f4aadbf44f7f523331774: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:05:16.246961 14641 consensus_queue.cc:1035] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:05:16.247116 14637 consensus_queue.cc:1035] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:05:16.249639 14644 mvcc.cc:205] Tried to move back new op lower bound from 7332737500142731264 to 7332737500108906496. Current Snapshot: MvccSnapshot[applied={T|T < 7332737500142731264}]
I20260924 03:05:16.249846 14646 mvcc.cc:205] Tried to move back new op lower bound from 7332737500142731264 to 7332737500108906496. Current Snapshot: MvccSnapshot[applied={T|T < 7332737500142731264}]
I20260924 03:05:16.252159 14154 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:05:16.252259 14154 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:05:16.253062 14154 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712: Sending DeleteTablet for 3 replicas of tablet 12dbf408ec95486982e3284a4b7a2740
I20260924 03:05:16.253412 14280 tablet_service.cc:1514] Processing DeleteTablet for tablet 12dbf408ec95486982e3284a4b7a2740 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:05:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57036
I20260924 03:05:16.253479 14548 tablet_service.cc:1514] Processing DeleteTablet for tablet 12dbf408ec95486982e3284a4b7a2740 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:05:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38562
I20260924 03:05:16.253671 14654 tablet_replica.cc:330] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774: stopping tablet replica
I20260924 03:05:16.253651 14414 tablet_service.cc:1514] Processing DeleteTablet for tablet 12dbf408ec95486982e3284a4b7a2740 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:05:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33092
I20260924 03:05:16.253765 14654 raft_consensus.cc:2229] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:05:16.253850 14654 raft_consensus.cc:2258] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:05:16.253849 14656 tablet_replica.cc:330] T 12dbf408ec95486982e3284a4b7a2740 P 3d5e3a3c37cd4aeb8c39f300acc4d502: stopping tablet replica
I20260924 03:05:16.253983 14656 raft_consensus.cc:2229] T 12dbf408ec95486982e3284a4b7a2740 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:05:16.254065 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 14217
I20260924 03:05:16.254089 14656 raft_consensus.cc:2258] T 12dbf408ec95486982e3284a4b7a2740 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:05:16.255184 14656 ts_tablet_manager.cc:1905] T 12dbf408ec95486982e3284a4b7a2740 P 3d5e3a3c37cd4aeb8c39f300acc4d502: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:05:16.255331 14655 tablet_replica.cc:330] T 12dbf408ec95486982e3284a4b7a2740 P be77a11fa86949e0a5c7cdf9af71f901: stopping tablet replica
I20260924 03:05:16.255451 14655 raft_consensus.cc:2229] T 12dbf408ec95486982e3284a4b7a2740 P be77a11fa86949e0a5c7cdf9af71f901 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:05:16.255560 14655 raft_consensus.cc:2258] T 12dbf408ec95486982e3284a4b7a2740 P be77a11fa86949e0a5c7cdf9af71f901 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:05:16.257016 14655 ts_tablet_manager.cc:1905] T 12dbf408ec95486982e3284a4b7a2740 P be77a11fa86949e0a5c7cdf9af71f901: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:05:16.257296 14656 ts_tablet_manager.cc:1918] T 12dbf408ec95486982e3284a4b7a2740 P 3d5e3a3c37cd4aeb8c39f300acc4d502: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:05:16.257375 14656 log.cc:1196] T 12dbf408ec95486982e3284a4b7a2740 P 3d5e3a3c37cd4aeb8c39f300acc4d502: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/wal/wals/12dbf408ec95486982e3284a4b7a2740
I20260924 03:05:16.257656 14656 ts_tablet_manager.cc:1939] T 12dbf408ec95486982e3284a4b7a2740 P 3d5e3a3c37cd4aeb8c39f300acc4d502: Deleting consensus metadata
I20260924 03:05:16.258213 14142 catalog_manager.cc:4841] TS 3d5e3a3c37cd4aeb8c39f300acc4d502 (127.0.98.2:34387): tablet 12dbf408ec95486982e3284a4b7a2740 (table test-table [id=af5d824447984492ba3a58735f29eb30]) successfully deleted
I20260924 03:05:16.258638 14655 ts_tablet_manager.cc:1918] T 12dbf408ec95486982e3284a4b7a2740 P be77a11fa86949e0a5c7cdf9af71f901: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:05:16.258720 14655 log.cc:1196] T 12dbf408ec95486982e3284a4b7a2740 P be77a11fa86949e0a5c7cdf9af71f901: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/wal/wals/12dbf408ec95486982e3284a4b7a2740
I20260924 03:05:16.258968 14655 ts_tablet_manager.cc:1939] T 12dbf408ec95486982e3284a4b7a2740 P be77a11fa86949e0a5c7cdf9af71f901: Deleting consensus metadata
I20260924 03:05:16.259526 14141 catalog_manager.cc:4841] TS be77a11fa86949e0a5c7cdf9af71f901 (127.0.98.3:35215): tablet 12dbf408ec95486982e3284a4b7a2740 (table test-table [id=af5d824447984492ba3a58735f29eb30]) successfully deleted
W20260924 03:05:16.260761 14140 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41695 (error 108)
I20260924 03:05:16.260895 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 14351
I20260924 03:05:16.261950 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.261794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41695 (local address 127.0.0.1:57058)
0924 03:05:16.261864 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:16.261908 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 03:05:16.262118 14140 catalog_manager.cc:4584] TS 47b995a4829f4aadbf44f7f523331774 (127.0.98.1:41695): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 12dbf408ec95486982e3284a4b7a2740: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41695: connect: Connection refused (error 111)
I20260924 03:05:16.266567 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 14485
I20260924 03:05:16.271651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 14123
I20260924 03:05:16.277261 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39953
--webserver_interface=127.0.98.62
--webserver_port=34447
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:05:16.362722 14658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:16.363194 14658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:16.363261 14658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:16.364712 14658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:05:16.364795 14658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:16.364823 14658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:16.364866 14658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:05:16.364889 14658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:05:16.366541 14658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:16.366858 14658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:16.367110 14658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:16.376668 14667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:16.376668 14665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:16.376669 14664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:16.377458 14658 server_base.cc:1034] running on GCE node
I20260924 03:05:16.377776 14658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:16.378140 14658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:16.379339 14658 hybrid_clock.cc:648] HybridClock initialized: now 1790219116379327 us; error 53 us; skew 500 ppm
Sep 24 03:05:16 dist-test-slave-wgsb krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:16.391661 14658 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:05:16.392298 14658 webserver.cc:460] Webserver started at http://127.0.98.62:34447/ using document root <none> and password file <none>
I20260924 03:05:16.392545 14658 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:16.392602 14658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:16.394052 14658 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:05:16.394829 14674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:16.395324 14658 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:05:16.395457 14658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "c5beaf31902c40e2baea7c9ea29b4712"
format_stamp: "Formatted at 2026-09-24 03:05:15 on dist-test-slave-wgsb"
server_key: "023839d15d37656f4b66540ccfe905fb"
server_key_iv: "88f7278b939ee3e63de490bc3a4baf9d"
server_key_version: "encryptionkey@0"
I20260924 03:05:16.395996 14658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:16.419293 14658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:16.422636 14658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:16.422915 14658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:16.427368 14658 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39953
I20260924 03:05:16.427421 14726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39953 every 8 connection(s)
I20260924 03:05:16.427870 14658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:05:16.429557 14727 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:05:16.430385 14727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712: Bootstrap starting.
I20260924 03:05:16.432778 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 14658
I20260924 03:05:16.433112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41695
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39953
--webserver_port=33163
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:05:16.439227 14727 log.cc:824] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712: Log is configured to *not* fsync() on all Append() calls
I20260924 03:05:16.441347 14727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:05:16.441665 14727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712: Bootstrap complete.
I20260924 03:05:16.444130 14727 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5beaf31902c40e2baea7c9ea29b4712" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39953 } }
I20260924 03:05:16.444394 14727 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5beaf31902c40e2baea7c9ea29b4712, State: Initialized, Role: FOLLOWER
I20260924 03:05:16.444586 14727 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5beaf31902c40e2baea7c9ea29b4712" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39953 } }
I20260924 03:05:16.444664 14727 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:05:16.444721 14727 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:05:16.444787 14727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:05:16.445891 14727 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5beaf31902c40e2baea7c9ea29b4712" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39953 } }
I20260924 03:05:16.446015 14727 leader_election.cc:302] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5beaf31902c40e2baea7c9ea29b4712; no voters:
I20260924 03:05:16.446228 14727 leader_election.cc:288] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 03:05:16.446271 14731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 03:05:16.446386 14731 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [term 2 LEADER]: Becoming Leader. State: Replica: c5beaf31902c40e2baea7c9ea29b4712, State: Running, Role: LEADER
I20260924 03:05:16.446488 14731 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5beaf31902c40e2baea7c9ea29b4712" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39953 } }
I20260924 03:05:16.446516 14727 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:05:16.446756 14731 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5beaf31902c40e2baea7c9ea29b4712. Latest consensus state: current_term: 2 leader_uuid: "c5beaf31902c40e2baea7c9ea29b4712" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5beaf31902c40e2baea7c9ea29b4712" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39953 } } }
I20260924 03:05:16.446856 14731 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [sys.catalog]: This master's current role is: LEADER
I20260924 03:05:16.447289 14735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:05:16.448101 14733 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c5beaf31902c40e2baea7c9ea29b4712" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5beaf31902c40e2baea7c9ea29b4712" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39953 } } }
I20260924 03:05:16.448179 14735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:05:16.448410 14735 catalog_manager.cc:1259] Loaded cluster ID: e0338a1bcc984f589a4ca3b9a163155d
I20260924 03:05:16.448280 14733 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [sys.catalog]: This master's current role is: LEADER
I20260924 03:05:16.448495 14735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:05:16.449048 14735 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 03:05:16.451785 14735 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 03:05:16.453852 14735 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:05:16.454128 14735 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712: Loaded TSK: 0
I20260924 03:05:16.454358 14735 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 03:05:16.538645 14729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:16.538914 14729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:16.538942 14729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:16.540241 14729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:16.540288 14729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:16.540318 14729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:05:16.541947 14729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41695
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33163
--tserver_master_addrs=127.0.98.62:39953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:16.542223 14729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:16.542477 14729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:16.548378 14755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:16.548418 14757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:16.548408 14729 server_base.cc:1034] running on GCE node
W20260924 03:05:16.548588 14754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:16.548913 14729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:16.549217 14729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:16.550407 14729 hybrid_clock.cc:648] HybridClock initialized: now 1790219116550390 us; error 31 us; skew 500 ppm
Sep 24 03:05:16 dist-test-slave-wgsb krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:16.553468 14729 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:05:16.554028 14729 webserver.cc:460] Webserver started at http://127.0.98.1:33163/ using document root <none> and password file <none>
I20260924 03:05:16.554275 14729 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:16.554355 14729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:16.557039 14729 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 03:05:16.557916 14764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:16.558254 14729 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:05:16.558370 14729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "47b995a4829f4aadbf44f7f523331774"
format_stamp: "Formatted at 2026-09-24 03:05:15 on dist-test-slave-wgsb"
server_key: "d0bb61df205658d4d361c4860cfe0048"
server_key_iv: "783c6e5b5382b3ec81e072acc1412f7a"
server_key_version: "encryptionkey@0"
I20260924 03:05:16.558779 14729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:16.587875 14729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:16.590318 14729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:16.590489 14729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:16.590735 14729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:05:16.591264 14771 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 03:05:16.594215 14729 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 03:05:16.594293 14729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260924 03:05:16.594323 14729 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 03:05:16.595083 14729 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 03:05:16.595129 14729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 03:05:16.595175 14771 tablet_bootstrap.cc:492] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774: Bootstrap starting.
I20260924 03:05:16.597301 14771 log.cc:824] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774: Log is configured to *not* fsync() on all Append() calls
I20260924 03:05:16.601040 14771 tablet_bootstrap.cc:492] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:05:16.601634 14771 tablet_bootstrap.cc:492] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774: Bootstrap complete.
I20260924 03:05:16.601872 14771 ts_tablet_manager.cc:1397] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260924 03:05:16.603163 14729 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41695
I20260924 03:05:16.603202 14879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41695 every 8 connection(s)
I20260924 03:05:16.603627 14729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:05:16.604305 14771 raft_consensus.cc:348] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } }
I20260924 03:05:16.604640 14771 raft_consensus.cc:729] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47b995a4829f4aadbf44f7f523331774, State: Initialized, Role: FOLLOWER
I20260924 03:05:16.604764 14771 consensus_queue.cc:260] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } }
I20260924 03:05:16.605024 14771 ts_tablet_manager.cc:1428] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:05:16.609872 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 14729
I20260924 03:05:16.610201 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34387
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39953
--webserver_port=44969
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:05:16 dist-test-slave-wgsb krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:05:16.617470 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.604356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.1:57247)
0924 03:05:16.604541 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:16.604589 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:16.604752 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:16.605516 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:16.605521 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:16.605858 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:16.606072 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:16.606077 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.607357 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.607359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:16.608273 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:16.608276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.608904 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.608906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:16.609055 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.609683 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:16.609694 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:16.612244 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:16.615559 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:16.615571 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:16.615573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:16.615951 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:16.616224 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:16.616225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:16.616225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:16.616361 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:16.616776 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:16.616783 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:16.616933 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 03:05:16.617039 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 03:05:16.617242 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":102,"threads_started":1}
I20260924 03:05:16.617453 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.604633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57247 (local address 127.0.98.62:39953)
0924 03:05:16.604989 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:16.604998 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:16.605031 (+ 33us) server_negotiation.cc:407] Connection header received
0924 03:05:16.605176 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:16.605182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:16.605231 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:16.605459 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:16.606202 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.607183 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.608386 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.608792 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.608915 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.612421 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:16.612427 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:16.612427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:16.612445 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:16.615464 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:16.616034 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:16.616037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:16.616037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:16.616147 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:16.616439 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:16.616441 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:16.616441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:16.616698 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:16.616801 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:16.617103 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 03:05:16.617242 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":80,"threads_started":1}
I20260924 03:05:16.618330 14880 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39953
I20260924 03:05:16.618484 14880 heartbeater.cc:464] Registering TS with master...
I20260924 03:05:16.618758 14880 heartbeater.cc:511] Master 127.0.98.62:39953 requested a full tablet report, sending...
I20260924 03:05:16.619488 14691 ts_manager.cc:194] Registered new tserver with Master: 47b995a4829f4aadbf44f7f523331774 (127.0.98.1:41695)
I20260924 03:05:16.620616 14691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57247
I20260924 03:05:16.621470 14880 heartbeater.cc:503] Master 127.0.98.62:39953 was elected leader, sending a full tablet report...
Sep 24 03:05:16 dist-test-slave-wgsb krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 03:05:16.630021 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.620151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41695 (local address 127.0.0.1:57070)
0924 03:05:16.620344 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:16.620379 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:16.620457 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:16.621310 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:16.621315 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:16.621835 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:16.621909 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:16.621912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.622741 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.622743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:16.623936 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:16.623940 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.624478 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.624482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:16.624567 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.625137 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:16.625142 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:16.626617 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:16.628721 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:16.628729 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:16.628731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:16.628953 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:16.629201 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:16.629202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:16.629203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:16.629265 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:16.629622 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:16.629628 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:16.629686 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 03:05:16.629779 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 03:05:16.629917 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":112,"threads_started":1}
I20260924 03:05:16.630034 14887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.620897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57070 (local address 127.0.98.1:41695)
0924 03:05:16.621120 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:16.621125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:16.621136 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:05:16.621174 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:16.621176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:16.621197 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:16.621274 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:16.621985 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.622656 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.624065 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.624399 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.624468 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.626744 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:16.626757 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:16.626757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:16.626764 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:16.628600 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:16.629059 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:16.629062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:16.629063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:16.629128 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:16.629338 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:16.629339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:16.629340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:16.629487 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:16.629583 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:16.629817 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 03:05:16.629909 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":120,"threads_started":1}
I20260924 03:05:16.630486 14814 tablet_service.cc:1514] Processing DeleteTablet for tablet 12dbf408ec95486982e3284a4b7a2740 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:05:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57070
I20260924 03:05:16.631181 14888 tablet_replica.cc:330] stopping tablet replica
I20260924 03:05:16.631315 14888 raft_consensus.cc:2229] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:05:16.631417 14888 raft_consensus.cc:2258] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:05:16.631801 14888 ts_tablet_manager.cc:1905] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:05:16.634438 14888 ts_tablet_manager.cc:1918] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:05:16.634526 14888 log.cc:1196] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/wal/wals/12dbf408ec95486982e3284a4b7a2740
I20260924 03:05:16.634756 14888 ts_tablet_manager.cc:1939] T 12dbf408ec95486982e3284a4b7a2740 P 47b995a4829f4aadbf44f7f523331774: Deleting consensus metadata
I20260924 03:05:16.635486 14676 catalog_manager.cc:4841] TS 47b995a4829f4aadbf44f7f523331774 (127.0.98.1:41695): tablet 12dbf408ec95486982e3284a4b7a2740 (table test-table [id=af5d824447984492ba3a58735f29eb30]) successfully deleted
W20260924 03:05:16.703249 14885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:16.703470 14885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:16.703495 14885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:16.704734 14885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:16.704783 14885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:16.704813 14885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:05:16.706338 14885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34387
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44969
--tserver_master_addrs=127.0.98.62:39953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:16.706604 14885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:16.706848 14885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:16.712417 14894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:16.712543 14885 server_base.cc:1034] running on GCE node
W20260924 03:05:16.712540 14893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:16.712390 14896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:16.712885 14885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:16.713104 14885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:16.714272 14885 hybrid_clock.cc:648] HybridClock initialized: now 1790219116714258 us; error 26 us; skew 500 ppm
Sep 24 03:05:16 dist-test-slave-wgsb krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:16.717119 14885 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:05:16.717553 14885 webserver.cc:460] Webserver started at http://127.0.98.2:44969/ using document root <none> and password file <none>
I20260924 03:05:16.717815 14885 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:16.717876 14885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:16.719514 14885 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:05:16.720325 14903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:16.720644 14885 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 03:05:16.720759 14885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502"
format_stamp: "Formatted at 2026-09-24 03:05:15 on dist-test-slave-wgsb"
server_key: "1d582927be7b38cf73874e300f1c9c5c"
server_key_iv: "dfa7fe97d9231262bcaf45a129a391ae"
server_key_version: "encryptionkey@0"
I20260924 03:05:16.721220 14885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:16.744338 14885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:16.747219 14885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:16.747401 14885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:16.747741 14885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:05:16.748131 14885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:05:16.748188 14885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:16.748234 14885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:05:16.748265 14885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:16.754750 14885 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34387
I20260924 03:05:16.754791 15016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34387 every 8 connection(s)
I20260924 03:05:16.755218 14885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:05:16.757712 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 14885
I20260924 03:05:16.758026 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35215
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39953
--webserver_port=46089
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:05:16 dist-test-slave-wgsb krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:05:16.766844 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.756093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45995 (local address 127.0.98.62:39953)
0924 03:05:16.756185 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:16.756188 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:16.756396 (+ 208us) server_negotiation.cc:407] Connection header received
0924 03:05:16.756879 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:16.756880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:16.756891 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:16.756983 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:16.758103 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.758972 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.760123 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.760512 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.760595 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.763400 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:16.763402 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:16.763403 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:16.763412 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:16.765132 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:16.765703 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:16.765704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:16.765705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:16.765774 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:16.766091 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:16.766092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:16.766092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:16.766230 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:16.766314 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:16.766634 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 03:05:16.766733 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 03:05:16.767084 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.756062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.2:45995)
0924 03:05:16.756259 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:16.756299 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:16.756784 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:16.757040 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:16.757043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:16.757668 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:16.757991 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:16.757998 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.759084 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.759085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:16.760025 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:16.760028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.760617 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.760618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:16.760770 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.761385 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:16.761395 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:16.763276 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:16.765266 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:16.765277 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:16.765280 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:16.765594 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:16.765856 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:16.765857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:16.765858 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:16.766005 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:16.766304 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:16.766312 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:16.766487 (+ 175us) client_negotiation.cc:769] Sending connection context
0924 03:05:16.766612 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 03:05:16.766807 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":111,"threads_started":1}
I20260924 03:05:16.767655 15017 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39953
I20260924 03:05:16.767759 15017 heartbeater.cc:464] Registering TS with master...
I20260924 03:05:16.767995 15017 heartbeater.cc:511] Master 127.0.98.62:39953 requested a full tablet report, sending...
I20260924 03:05:16.768462 14691 ts_manager.cc:194] Registered new tserver with Master: 3d5e3a3c37cd4aeb8c39f300acc4d502 (127.0.98.2:34387)
I20260924 03:05:16.769037 14691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45995
W20260924 03:05:16.844561 15020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:16.844769 15020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:16.844792 15020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:16.845913 15020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:16.845952 15020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:16.845978 15020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:05:16.847640 15020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46089
--tserver_master_addrs=127.0.98.62:39953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:16.847844 15020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:16.848073 15020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:16.853427 15026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:16.853449 15028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:16.853515 15025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:16.853741 15020 server_base.cc:1034] running on GCE node
I20260924 03:05:16.853967 15020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:16.854189 15020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:16.855324 15020 hybrid_clock.cc:648] HybridClock initialized: now 1790219116855306 us; error 37 us; skew 500 ppm
Sep 24 03:05:16 dist-test-slave-wgsb krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:16.857928 15020 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:05:16.858409 15020 webserver.cc:460] Webserver started at http://127.0.98.3:46089/ using document root <none> and password file <none>
I20260924 03:05:16.858615 15020 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:16.858670 15020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:16.860208 15020 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:05:16.860786 15035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:16.861028 15020 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:05:16.861109 15020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "be77a11fa86949e0a5c7cdf9af71f901"
format_stamp: "Formatted at 2026-09-24 03:05:16 on dist-test-slave-wgsb"
server_key: "9721af1ac5b03b5969da67ae425ee356"
server_key_iv: "74f650090565e0613c1136ac3c1bc00f"
server_key_version: "encryptionkey@0"
I20260924 03:05:16.861471 15020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:16.868765 15020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:16.870579 15020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:16.870690 15020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:16.870867 15020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:05:16.871105 15020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:05:16.871134 15020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:16.871156 15020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:05:16.871169 15020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:16.876822 15020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35215
I20260924 03:05:16.876895 15148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35215 every 8 connection(s)
I20260924 03:05:16.877254 15020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 03:05:16 dist-test-slave-wgsb krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:05:16.884382 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 15020
I20260924 03:05:16.887032 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.878105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49429 (local address 127.0.98.62:39953)
0924 03:05:16.878193 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:16.878195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:16.878416 (+ 221us) server_negotiation.cc:407] Connection header received
0924 03:05:16.878870 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:16.878871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:16.878882 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:16.878973 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:16.879956 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.880481 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.881498 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.881695 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.881751 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.883986 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:16.883988 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:16.883988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:16.883994 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:16.885586 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:16.886016 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:16.886017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:16.886018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:16.886082 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:16.886372 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:16.886374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:16.886374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:16.886492 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:16.886569 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:16.886852 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 03:05:16.886942 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 03:05:16.887221 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.878084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.3:49429)
0924 03:05:16.878282 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:16.878318 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:16.878786 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:16.879026 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:16.879029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:16.879582 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:16.879843 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:16.879847 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.880568 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.880569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:16.881417 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:16.881419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.881767 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.881768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:16.881905 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.882379 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:16.882389 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:16.883891 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:16.885666 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:16.885674 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:16.885676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:16.885919 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:16.886163 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:16.886164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:16.886165 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:16.886292 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:16.886560 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:16.886568 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:16.886712 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 03:05:16.886817 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 03:05:16.886974 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":114,"threads_started":1}
I20260924 03:05:16.887836 15149 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39953
I20260924 03:05:16.887939 15149 heartbeater.cc:464] Registering TS with master...
I20260924 03:05:16.888135 15149 heartbeater.cc:511] Master 127.0.98.62:39953 requested a full tablet report, sending...
I20260924 03:05:16.888614 14691 ts_manager.cc:194] Registered new tserver with Master: be77a11fa86949e0a5c7cdf9af71f901 (127.0.98.3:35215)
I20260924 03:05:16.889039 14691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49429
I20260924 03:05:16.892040 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.884799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35940 (local address 127.0.98.62:39953)
0924 03:05:16.884953 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:16.884955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:16.885108 (+ 153us) server_negotiation.cc:407] Connection header received
0924 03:05:16.885171 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:16.885171 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:16.885186 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:16.885255 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:16.885896 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.886442 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.887540 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.887903 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.887995 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.889366 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:16.889368 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:16.889368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:16.889376 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:16.890896 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:16.891320 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:16.891321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:16.891321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:16.891378 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:16.891549 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:16.891549 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:16.891550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:16.891700 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:16.891874 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:16.891914 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 03:05:16.891978 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":72,"threads_started":1}
I20260924 03:05:16.892853 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:05:16.895581 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.893213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35954 (local address 127.0.98.62:39953)
0924 03:05:16.893251 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:16.893253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:16.893389 (+ 136us) server_negotiation.cc:407] Connection header received
0924 03:05:16.893429 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:16.893430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:16.893454 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:16.893512 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:05:16.893658 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.894096 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.894890 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.895076 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.895133 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.895267 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:05:16.895348 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:05:16.895409 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:16.895477 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 03:05:16.895494 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:05:16.895895 14691 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:05:16.897527 14691 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:05:16.900141 14814 tablet_service.cc:1467] Processing CreateTablet for tablet 2d35faf1f16149d09aec08ff090c93f0 (DEFAULT_TABLE table=test-table [id=2f7b822ecd38491694efe4c8d5b4f422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:05:16.900367 14814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d35faf1f16149d09aec08ff090c93f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:05:16.902014 14771 tablet_bootstrap.cc:492] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774: Bootstrap starting.
I20260924 03:05:16.902405 14771 tablet_bootstrap.cc:654] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774: Neither blocks nor log segments found. Creating new log.
I20260924 03:05:16.903698 14771 tablet_bootstrap.cc:492] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774: No bootstrap required, opened a new log
I20260924 03:05:16.903760 14771 ts_tablet_manager.cc:1397] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:05:16.904059 14771 raft_consensus.cc:348] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } }
I20260924 03:05:16.904152 14771 raft_consensus.cc:374] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:05:16.904213 14771 raft_consensus.cc:729] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47b995a4829f4aadbf44f7f523331774, State: Initialized, Role: FOLLOWER
I20260924 03:05:16.904268 14771 consensus_queue.cc:260] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } }
I20260924 03:05:16.904382 14771 ts_tablet_manager.cc:1428] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 03:05:16.904561 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.899760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35215 (local address 127.0.0.1:38566)
0924 03:05:16.900051 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:16.900063 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:16.900120 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:16.900337 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:16.900338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:16.900341 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:05:16.900405 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:16.900407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.901762 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.901763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:16.903119 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:16.903122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.904274 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.904275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:16.904363 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.904397 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 03:05:16.904435 (+ 38us) client_negotiation.cc:240] Negotiation successful
0924 03:05:16.904494 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274}
I20260924 03:05:16.904882 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.899791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34387 (local address 127.0.0.1:33098)
0924 03:05:16.900452 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:16.900464 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:16.900516 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:16.900886 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:16.900887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:16.900890 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:05:16.900933 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:16.900935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.902187 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.902187 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:16.903458 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:16.903461 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.904608 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.904609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:16.904697 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.904702 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:05:16.904750 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 03:05:16.904797 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":76,"threads_started":1}
I20260924 03:05:16.904985 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.899854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38566 (local address 127.0.98.3:35215)
0924 03:05:16.900029 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:16.900034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:16.900111 (+ 77us) server_negotiation.cc:407] Connection header received
0924 03:05:16.900192 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:16.900193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:16.900206 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:16.900508 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:05:16.900576 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.901642 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.903257 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.904158 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.904743 (+ 585us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.904816 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:16.904845 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 03:05:16.904897 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":108,"threads_started":1}
I20260924 03:05:16.905432 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:16.900232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33098 (local address 127.0.98.2:34387)
0924 03:05:16.900648 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:16.900653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:16.900663 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:05:16.900703 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:16.900705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:16.900722 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:16.900801 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:05:16.901018 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.902011 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.903560 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:16.904308 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:16.904385 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:16.905264 (+ 879us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:16.905292 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 03:05:16.905342 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":99,"threads_started":1}
I20260924 03:05:16.905668 15083 tablet_service.cc:1467] Processing CreateTablet for tablet 2d35faf1f16149d09aec08ff090c93f0 (DEFAULT_TABLE table=test-table [id=2f7b822ecd38491694efe4c8d5b4f422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:05:16.906044 15083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d35faf1f16149d09aec08ff090c93f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:05:16.906164 14951 tablet_service.cc:1467] Processing CreateTablet for tablet 2d35faf1f16149d09aec08ff090c93f0 (DEFAULT_TABLE table=test-table [id=2f7b822ecd38491694efe4c8d5b4f422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:05:16.906486 14951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d35faf1f16149d09aec08ff090c93f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:05:16.909065 15159 tablet_bootstrap.cc:492] T 2d35faf1f16149d09aec08ff090c93f0 P be77a11fa86949e0a5c7cdf9af71f901: Bootstrap starting.
I20260924 03:05:16.909767 15160 tablet_bootstrap.cc:492] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502: Bootstrap starting.
I20260924 03:05:16.909859 15159 tablet_bootstrap.cc:654] T 2d35faf1f16149d09aec08ff090c93f0 P be77a11fa86949e0a5c7cdf9af71f901: Neither blocks nor log segments found. Creating new log.
I20260924 03:05:16.910384 15160 tablet_bootstrap.cc:654] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502: Neither blocks nor log segments found. Creating new log.
I20260924 03:05:16.910395 15159 log.cc:824] T 2d35faf1f16149d09aec08ff090c93f0 P be77a11fa86949e0a5c7cdf9af71f901: Log is configured to *not* fsync() on all Append() calls
I20260924 03:05:16.910882 15160 log.cc:824] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502: Log is configured to *not* fsync() on all Append() calls
I20260924 03:05:16.911235 15159 tablet_bootstrap.cc:492] T 2d35faf1f16149d09aec08ff090c93f0 P be77a11fa86949e0a5c7cdf9af71f901: No bootstrap required, opened a new log
I20260924 03:05:16.911346 15159 ts_tablet_manager.cc:1397] T 2d35faf1f16149d09aec08ff090c93f0 P be77a11fa86949e0a5c7cdf9af71f901: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:05:16.911777 15160 tablet_bootstrap.cc:492] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502: No bootstrap required, opened a new log
I20260924 03:05:16.911942 15160 ts_tablet_manager.cc:1397] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:05:16.913079 15159 raft_consensus.cc:348] T 2d35faf1f16149d09aec08ff090c93f0 P be77a11fa86949e0a5c7cdf9af71f901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } }
I20260924 03:05:16.913233 15159 raft_consensus.cc:374] T 2d35faf1f16149d09aec08ff090c93f0 P be77a11fa86949e0a5c7cdf9af71f901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:05:16.913259 15159 raft_consensus.cc:729] T 2d35faf1f16149d09aec08ff090c93f0 P be77a11fa86949e0a5c7cdf9af71f901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be77a11fa86949e0a5c7cdf9af71f901, State: Initialized, Role: FOLLOWER
I20260924 03:05:16.913408 15159 consensus_queue.cc:260] T 2d35faf1f16149d09aec08ff090c93f0 P be77a11fa86949e0a5c7cdf9af71f901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } }
I20260924 03:05:16.913352 15160 raft_consensus.cc:348] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } }
I20260924 03:05:16.913561 15160 raft_consensus.cc:374] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:05:16.913609 15160 raft_consensus.cc:729] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d5e3a3c37cd4aeb8c39f300acc4d502, State: Initialized, Role: FOLLOWER
I20260924 03:05:16.913725 15160 consensus_queue.cc:260] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } }
I20260924 03:05:16.913735 15159 ts_tablet_manager.cc:1428] T 2d35faf1f16149d09aec08ff090c93f0 P be77a11fa86949e0a5c7cdf9af71f901: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 03:05:16.913735 15149 heartbeater.cc:503] Master 127.0.98.62:39953 was elected leader, sending a full tablet report...
I20260924 03:05:16.914029 15160 ts_tablet_manager.cc:1428] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:05:16.914191 15017 heartbeater.cc:503] Master 127.0.98.62:39953 was elected leader, sending a full tablet report...
W20260924 03:05:17.005936 15018 tablet.cc:2369] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:05:17.085618 15164 raft_consensus.cc:482] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:05:17.085745 15164 raft_consensus.cc:504] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } }
I20260924 03:05:17.086052 15164 leader_election.cc:288] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be77a11fa86949e0a5c7cdf9af71f901 (127.0.98.3:35215), 47b995a4829f4aadbf44f7f523331774 (127.0.98.1:41695)
I20260924 03:05:17.090464 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:17.086451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44957 (local address 127.0.98.3:35215)
0924 03:05:17.086518 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:17.086520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:17.086535 (+ 15us) server_negotiation.cc:407] Connection header received
0924 03:05:17.086560 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:17.086561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:17.086570 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:17.086627 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:05:17.086815 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:17.087667 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:17.089374 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:17.090031 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:17.090116 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:17.090128 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:17.090346 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 03:05:17.090391 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 03:05:17.090464 14887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:17.086341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34513 (local address 127.0.98.1:41695)
0924 03:05:17.086425 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:17.086427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:17.086715 (+ 288us) server_negotiation.cc:407] Connection header received
0924 03:05:17.086740 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:17.086741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:17.086763 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:17.086843 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:05:17.087001 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:17.087744 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:17.089373 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:17.090045 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:17.090116 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:17.090201 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:17.090346 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 03:05:17.090394 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 03:05:17.090471 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:17.086275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35215 (local address 127.0.98.2:44957)
0924 03:05:17.086324 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:17.086336 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:17.086380 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:17.086659 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:17.086660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:17.086662 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:05:17.086737 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:17.086739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:17.087852 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:17.087853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:17.089232 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:17.089236 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:17.090150 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:17.090151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:17.090253 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:17.090254 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 03:05:17.090321 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 03:05:17.090370 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":59}
I20260924 03:05:17.090476 15165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:17.086275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41695 (local address 127.0.98.2:34513)
0924 03:05:17.086614 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:17.086626 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:17.086684 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:17.086865 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:17.086866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:17.086867 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:05:17.086924 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:17.086925 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:17.087852 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:17.087853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:17.089232 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:17.089236 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:17.090150 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:17.090151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:17.090248 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:17.090253 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:05:17.090321 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 03:05:17.090372 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":257,"thread_start_us":127,"threads_started":1}
I20260924 03:05:17.090924 15103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d35faf1f16149d09aec08ff090c93f0" candidate_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be77a11fa86949e0a5c7cdf9af71f901" is_pre_election: true
I20260924 03:05:17.090916 14834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d35faf1f16149d09aec08ff090c93f0" candidate_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47b995a4829f4aadbf44f7f523331774" is_pre_election: true
I20260924 03:05:17.091095 14834 raft_consensus.cc:2454] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d5e3a3c37cd4aeb8c39f300acc4d502 in term 0.
I20260924 03:05:17.091112 15103 raft_consensus.cc:2454] T 2d35faf1f16149d09aec08ff090c93f0 P be77a11fa86949e0a5c7cdf9af71f901 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d5e3a3c37cd4aeb8c39f300acc4d502 in term 0.
I20260924 03:05:17.091354 14905 leader_election.cc:302] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d5e3a3c37cd4aeb8c39f300acc4d502, 47b995a4829f4aadbf44f7f523331774; no voters:
I20260924 03:05:17.091506 15164 raft_consensus.cc:2781] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:05:17.091562 15164 raft_consensus.cc:482] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:05:17.091605 15164 raft_consensus.cc:3037] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:05:17.092329 15164 raft_consensus.cc:504] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } }
I20260924 03:05:17.092476 15164 leader_election.cc:288] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [CANDIDATE]: Term 1 election: Requested vote from peers be77a11fa86949e0a5c7cdf9af71f901 (127.0.98.3:35215), 47b995a4829f4aadbf44f7f523331774 (127.0.98.1:41695)
I20260924 03:05:17.092615 15103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d35faf1f16149d09aec08ff090c93f0" candidate_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be77a11fa86949e0a5c7cdf9af71f901"
I20260924 03:05:17.092648 14834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d35faf1f16149d09aec08ff090c93f0" candidate_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47b995a4829f4aadbf44f7f523331774"
I20260924 03:05:17.092715 15103 raft_consensus.cc:3037] T 2d35faf1f16149d09aec08ff090c93f0 P be77a11fa86949e0a5c7cdf9af71f901 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:05:17.092756 14834 raft_consensus.cc:3037] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:05:17.093405 14834 raft_consensus.cc:2454] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d5e3a3c37cd4aeb8c39f300acc4d502 in term 1.
I20260924 03:05:17.093572 14905 leader_election.cc:302] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d5e3a3c37cd4aeb8c39f300acc4d502, 47b995a4829f4aadbf44f7f523331774; no voters:
I20260924 03:05:17.093621 15103 raft_consensus.cc:2454] T 2d35faf1f16149d09aec08ff090c93f0 P be77a11fa86949e0a5c7cdf9af71f901 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d5e3a3c37cd4aeb8c39f300acc4d502 in term 1.
I20260924 03:05:17.093720 15164 raft_consensus.cc:2781] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:05:17.093899 15164 raft_consensus.cc:686] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [term 1 LEADER]: Becoming Leader. State: Replica: 3d5e3a3c37cd4aeb8c39f300acc4d502, State: Running, Role: LEADER
I20260924 03:05:17.094003 15164 consensus_queue.cc:237] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } }
I20260924 03:05:17.094846 14691 catalog_manager.cc:5495] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d5e3a3c37cd4aeb8c39f300acc4d502 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } health_report { overall_health: UNKNOWN } } }
W20260924 03:05:17.104578 14881 tablet.cc:2369] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:05:17.127856 15150 tablet.cc:2369] T 2d35faf1f16149d09aec08ff090c93f0 P be77a11fa86949e0a5c7cdf9af71f901: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:05:17.133455 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:17.130681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33108 (local address 127.0.98.2:34387)
0924 03:05:17.130762 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:17.130764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:17.130775 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:05:17.130930 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:17.130932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:17.130964 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:17.131056 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:05:17.131227 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:17.131804 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:17.132574 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:17.132765 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:17.132832 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:17.133036 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:05:17.133145 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:05:17.133206 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:17.133258 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 03:05:17.133277 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 03:05:17.135386 14834 raft_consensus.cc:1261] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [term 1 FOLLOWER]: Refusing update from remote peer 3d5e3a3c37cd4aeb8c39f300acc4d502: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:05:17.135444 15103 raft_consensus.cc:1261] T 2d35faf1f16149d09aec08ff090c93f0 P be77a11fa86949e0a5c7cdf9af71f901 [term 1 FOLLOWER]: Refusing update from remote peer 3d5e3a3c37cd4aeb8c39f300acc4d502: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:05:17.135861 15166 consensus_queue.cc:1035] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:05:17.136046 15164 consensus_queue.cc:1035] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:05:17.138096 15169 mvcc.cc:205] Tried to move back new op lower bound from 7332737503783768064 to 7332737503617523712. Current Snapshot: MvccSnapshot[applied={T|T < 7332737503783768064}]
I20260924 03:05:17.138669 15171 mvcc.cc:205] Tried to move back new op lower bound from 7332737503783768064 to 7332737503617523712. Current Snapshot: MvccSnapshot[applied={T|T < 7332737503783768064}]
I20260924 03:05:17.140986 14691 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:35954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:05:17.141098 14691 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:35954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:05:17.141865 14691 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712: Sending DeleteTablet for 3 replicas of tablet 2d35faf1f16149d09aec08ff090c93f0
I20260924 03:05:17.142192 14951 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d35faf1f16149d09aec08ff090c93f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:05:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33098
I20260924 03:05:17.142432 15083 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d35faf1f16149d09aec08ff090c93f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:05:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38566
I20260924 03:05:17.142453 15178 tablet_replica.cc:330] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502: stopping tablet replica
I20260924 03:05:17.142593 15178 raft_consensus.cc:2229] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:05:17.142783 15178 raft_consensus.cc:2258] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:05:17.142777 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 14729
I20260924 03:05:17.142784 14814 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d35faf1f16149d09aec08ff090c93f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:05:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57070
I20260924 03:05:17.143749 15179 tablet_replica.cc:330] T 2d35faf1f16149d09aec08ff090c93f0 P be77a11fa86949e0a5c7cdf9af71f901: stopping tablet replica
I20260924 03:05:17.143860 15179 raft_consensus.cc:2229] T 2d35faf1f16149d09aec08ff090c93f0 P be77a11fa86949e0a5c7cdf9af71f901 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:05:17.143962 15179 raft_consensus.cc:2258] T 2d35faf1f16149d09aec08ff090c93f0 P be77a11fa86949e0a5c7cdf9af71f901 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:05:17.144349 15179 ts_tablet_manager.cc:1905] T 2d35faf1f16149d09aec08ff090c93f0 P be77a11fa86949e0a5c7cdf9af71f901: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:05:17.145912 15179 ts_tablet_manager.cc:1918] T 2d35faf1f16149d09aec08ff090c93f0 P be77a11fa86949e0a5c7cdf9af71f901: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:05:17.145975 15179 log.cc:1196] T 2d35faf1f16149d09aec08ff090c93f0 P be77a11fa86949e0a5c7cdf9af71f901: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/wal/wals/2d35faf1f16149d09aec08ff090c93f0
I20260924 03:05:17.146283 15179 ts_tablet_manager.cc:1939] T 2d35faf1f16149d09aec08ff090c93f0 P be77a11fa86949e0a5c7cdf9af71f901: Deleting consensus metadata
I20260924 03:05:17.146684 14677 catalog_manager.cc:4841] TS be77a11fa86949e0a5c7cdf9af71f901 (127.0.98.3:35215): tablet 2d35faf1f16149d09aec08ff090c93f0 (table test-table [id=2f7b822ecd38491694efe4c8d5b4f422]) successfully deleted
I20260924 03:05:17.148697 15178 ts_tablet_manager.cc:1905] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 03:05:17.149849 14676 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41695 (error 108)
I20260924 03:05:17.150132 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 14885
I20260924 03:05:17.150393 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:17.150264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41695 (local address 127.0.0.1:57082)
0924 03:05:17.150317 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:17.150350 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 03:05:17.150609 14676 catalog_manager.cc:4584] TS 47b995a4829f4aadbf44f7f523331774 (127.0.98.1:41695): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d35faf1f16149d09aec08ff090c93f0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41695: connect: Connection refused (error 111)
I20260924 03:05:17.155471 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:17.155332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34387 (local address 127.0.0.1:33124)
0924 03:05:17.155395 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:17.155433 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 03:05:17.155496 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 15020
I20260924 03:05:17.160220 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 14658
I20260924 03:05:17.166213 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39953
--webserver_interface=127.0.98.62
--webserver_port=34447
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:05:17.253225 15181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:17.253508 15181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:17.253546 15181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:17.254760 15181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:05:17.254845 15181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:17.254869 15181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:17.255139 15181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:05:17.255177 15181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:05:17.257584 15181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:17.257907 15181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:17.258194 15181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:05:17.263485 15181 server_base.cc:1034] running on GCE node
W20260924 03:05:17.263449 15189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:17.263427 15187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:17.263428 15186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:17.264397 15181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:17.265087 15181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:17.266311 15181 hybrid_clock.cc:648] HybridClock initialized: now 1790219117266279 us; error 76 us; skew 500 ppm
Sep 24 03:05:17 dist-test-slave-wgsb krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:17.269564 15181 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:05:17.270154 15181 webserver.cc:460] Webserver started at http://127.0.98.62:34447/ using document root <none> and password file <none>
I20260924 03:05:17.270442 15181 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:17.270517 15181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:17.272073 15181 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:05:17.272897 15196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:17.273196 15181 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:05:17.273320 15181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "c5beaf31902c40e2baea7c9ea29b4712"
format_stamp: "Formatted at 2026-09-24 03:05:15 on dist-test-slave-wgsb"
server_key: "023839d15d37656f4b66540ccfe905fb"
server_key_iv: "88f7278b939ee3e63de490bc3a4baf9d"
server_key_version: "encryptionkey@0"
I20260924 03:05:17.273697 15181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:17.288559 15181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:17.290992 15181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:17.291172 15181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:17.295710 15181 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39953
I20260924 03:05:17.295776 15248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39953 every 8 connection(s)
I20260924 03:05:17.296212 15181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:05:17.297930 15249 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:05:17.298517 15249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712: Bootstrap starting.
I20260924 03:05:17.302100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 15181
I20260924 03:05:17.302444 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41695
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39953
--webserver_port=33163
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:05:17.308699 15249 log.cc:824] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712: Log is configured to *not* fsync() on all Append() calls
I20260924 03:05:17.312237 15249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:05:17.312662 15249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712: Bootstrap complete.
I20260924 03:05:17.316291 15249 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5beaf31902c40e2baea7c9ea29b4712" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39953 } }
I20260924 03:05:17.316617 15249 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5beaf31902c40e2baea7c9ea29b4712, State: Initialized, Role: FOLLOWER
I20260924 03:05:17.316776 15249 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5beaf31902c40e2baea7c9ea29b4712" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39953 } }
I20260924 03:05:17.316841 15249 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:05:17.316879 15249 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:05:17.316946 15249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [term 2 FOLLOWER]: Advancing to term 3
I20260924 03:05:17.318246 15249 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5beaf31902c40e2baea7c9ea29b4712" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39953 } }
I20260924 03:05:17.318353 15249 leader_election.cc:302] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5beaf31902c40e2baea7c9ea29b4712; no voters:
I20260924 03:05:17.318524 15249 leader_election.cc:288] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 03:05:17.318606 15253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 03:05:17.318763 15249 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:05:17.318770 15253 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [term 3 LEADER]: Becoming Leader. State: Replica: c5beaf31902c40e2baea7c9ea29b4712, State: Running, Role: LEADER
I20260924 03:05:17.318982 15253 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5beaf31902c40e2baea7c9ea29b4712" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39953 } }
I20260924 03:05:17.319262 15253 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5beaf31902c40e2baea7c9ea29b4712. Latest consensus state: current_term: 3 leader_uuid: "c5beaf31902c40e2baea7c9ea29b4712" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5beaf31902c40e2baea7c9ea29b4712" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39953 } } }
I20260924 03:05:17.319360 15253 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [sys.catalog]: This master's current role is: LEADER
I20260924 03:05:17.319262 15254 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c5beaf31902c40e2baea7c9ea29b4712" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5beaf31902c40e2baea7c9ea29b4712" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39953 } } }
I20260924 03:05:17.319466 15254 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5beaf31902c40e2baea7c9ea29b4712 [sys.catalog]: This master's current role is: LEADER
I20260924 03:05:17.319563 15262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:05:17.320446 15262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:05:17.320556 15262 catalog_manager.cc:1259] Loaded cluster ID: e0338a1bcc984f589a4ca3b9a163155d
I20260924 03:05:17.320600 15262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 03:05:17.320859 15262 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f55d9b949ad google::LogMessage::Fail() at ??:0
@ 0x7f55d9b9788a google::LogMessage::SendToLog() at ??:0
@ 0x7f55d9b944de google::LogMessage::Flush() at ??:0
@ 0x7f55d9b95a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f55e1ab50f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f55e1abe8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f55dba620d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f55dba57954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f55dc8bb6db start_thread at ??:0
@ 0x7f55d869261f clone at ??:0
W20260924 03:05:17.408607 15251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:17.408808 15251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:17.408835 15251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:17.410034 15251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:17.410079 15251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:17.410106 15251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:05:17.411576 15251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41695
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33163
--tserver_master_addrs=127.0.98.62:39953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:17.411895 15251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:17.412132 15251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:17.417366 15288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:17.417392 15286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:17.417584 15251 server_base.cc:1034] running on GCE node
W20260924 03:05:17.417682 15285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:17.417868 15251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:17.418572 15251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:17.419818 15251 hybrid_clock.cc:648] HybridClock initialized: now 1790219117419794 us; error 32 us; skew 500 ppm
Sep 24 03:05:17 dist-test-slave-wgsb krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:17.422626 15251 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:05:17.423175 15251 webserver.cc:460] Webserver started at http://127.0.98.1:33163/ using document root <none> and password file <none>
I20260924 03:05:17.423419 15251 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:17.423491 15251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:17.425019 15251 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:05:17.425866 15295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:17.426066 15251 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:05:17.426146 15251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "47b995a4829f4aadbf44f7f523331774"
format_stamp: "Formatted at 2026-09-24 03:05:15 on dist-test-slave-wgsb"
server_key: "d0bb61df205658d4d361c4860cfe0048"
server_key_iv: "783c6e5b5382b3ec81e072acc1412f7a"
server_key_version: "encryptionkey@0"
I20260924 03:05:17.426524 15251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:17.439284 15251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:17.442194 15251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:17.442345 15251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:17.442619 15251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:05:17.443151 15302 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 03:05:17.445240 15251 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 03:05:17.445317 15251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 03:05:17.445360 15251 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 03:05:17.446071 15251 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 03:05:17.446131 15251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 03:05:17.446208 15302 tablet_bootstrap.cc:492] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774: Bootstrap starting.
I20260924 03:05:17.453673 15251 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41695
I20260924 03:05:17.453827 15409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41695 every 8 connection(s)
I20260924 03:05:17.454133 15251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:05:17.458182 15302 log.cc:824] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774: Log is configured to *not* fsync() on all Append() calls
I20260924 03:05:17.460906 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 15251
I20260924 03:05:17.462157 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34387
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39953
--webserver_port=44969
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:05:17.466459 15302 tablet_bootstrap.cc:492] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:05:17.467280 15302 tablet_bootstrap.cc:492] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774: Bootstrap complete.
I20260924 03:05:17.468680 15302 ts_tablet_manager.cc:1397] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.007s
I20260924 03:05:17.473156 15302 raft_consensus.cc:348] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } }
I20260924 03:05:17.477748 15302 raft_consensus.cc:729] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47b995a4829f4aadbf44f7f523331774, State: Initialized, Role: FOLLOWER
I20260924 03:05:17.478086 15302 consensus_queue.cc:260] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } }
I20260924 03:05:17.478513 15302 ts_tablet_manager.cc:1428] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774: Time spent starting tablet: real 0.010s user 0.009s sys 0.002s
W20260924 03:05:17.596837 15414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:17.597112 15414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:17.597182 15414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:17.598460 15412 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:05:17.454811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.1:57993)
0924 03:05:17.455014 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:17.455060 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:17.455562 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:17.598259 (+142697us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":114,"threads_started":1}
W20260924 03:05:17.598695 15414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:17.598742 15414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:17.598784 15414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
W20260924 03:05:17.598835 15410 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39953 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39953: Client connection negotiation failed: client connection to 127.0.98.62:39953: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 03:05:17.599237 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:17.599102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.1:55925)
0924 03:05:17.599162 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:17.599210 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 03:05:17.599570 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:17.599473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.1:57589)
0924 03:05:17.599521 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:17.599548 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 03:05:17.599710 15410 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 03:05:17.600656 15414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34387
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44969
--tserver_master_addrs=127.0.98.62:39953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:17.600903 15414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:17.601154 15414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:17.606223 15421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:17.606267 15423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:17.606267 15420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:17.606303 15414 server_base.cc:1034] running on GCE node
I20260924 03:05:17.606595 15414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:17.606794 15414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:17.607928 15414 hybrid_clock.cc:648] HybridClock initialized: now 1790219117607916 us; error 28 us; skew 500 ppm
Sep 24 03:05:17 dist-test-slave-wgsb krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:17.610401 15414 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:05:17.610862 15414 webserver.cc:460] Webserver started at http://127.0.98.2:44969/ using document root <none> and password file <none>
I20260924 03:05:17.611099 15414 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:17.611168 15414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:17.612743 15414 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:05:17.613468 15430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:17.613655 15414 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:05:17.613747 15414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502"
format_stamp: "Formatted at 2026-09-24 03:05:15 on dist-test-slave-wgsb"
server_key: "1d582927be7b38cf73874e300f1c9c5c"
server_key_iv: "dfa7fe97d9231262bcaf45a129a391ae"
server_key_version: "encryptionkey@0"
I20260924 03:05:17.614076 15414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:17.634521 15414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:17.636739 15414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:17.636860 15414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:17.637072 15414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:05:17.637535 15437 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260924 03:05:17.639024 15437 ts_tablet_manager.cc:1852] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 03:05:17.639297 15437 ts_tablet_manager.cc:1905] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:05:17.642374 15437 ts_tablet_manager.cc:1918] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:05:17.642463 15437 log.cc:1196] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/wal/wals/2d35faf1f16149d09aec08ff090c93f0
I20260924 03:05:17.642817 15437 ts_tablet_manager.cc:1939] T 2d35faf1f16149d09aec08ff090c93f0 P 3d5e3a3c37cd4aeb8c39f300acc4d502: Deleting consensus metadata
I20260924 03:05:17.643074 15414 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 03:05:17.643168 15414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260924 03:05:17.643217 15414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:05:17.643245 15414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:17.649402 15414 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34387
I20260924 03:05:17.649462 15544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34387 every 8 connection(s)
I20260924 03:05:17.649753 15414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:05:17.650794 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:17.650425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.2:36691)
0924 03:05:17.650592 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:17.650662 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
W20260924 03:05:17.651064 15545 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39953 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39953: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
I20260924 03:05:17.651347 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:17.651228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.2:56883)
0924 03:05:17.651293 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:17.651324 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 03:05:17.651691 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:17.651575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.2:38603)
0924 03:05:17.651644 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:17.651674 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 03:05:17.651847 15545 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 03:05:17.653362 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 15414
I20260924 03:05:17.653743 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35215
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39953
--webserver_port=46089
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:05:17.741874 15548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:17.742113 15548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:17.742154 15548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:17.743435 15548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:17.743600 15548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:17.743748 15548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:05:17.745405 15548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46089
--tserver_master_addrs=127.0.98.62:39953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:17.745677 15548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:17.745926 15548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:17.751209 15554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:17.751209 15553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:17.751231 15556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:17.751672 15548 server_base.cc:1034] running on GCE node
I20260924 03:05:17.751873 15548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:17.752117 15548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:17.753270 15548 hybrid_clock.cc:648] HybridClock initialized: now 1790219117753240 us; error 43 us; skew 500 ppm
Sep 24 03:05:17 dist-test-slave-wgsb krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:17.755856 15548 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:05:17.756403 15548 webserver.cc:460] Webserver started at http://127.0.98.3:46089/ using document root <none> and password file <none>
I20260924 03:05:17.756628 15548 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:17.756700 15548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:17.758081 15548 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:05:17.758749 15563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:17.759160 15548 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:05:17.759316 15548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "be77a11fa86949e0a5c7cdf9af71f901"
format_stamp: "Formatted at 2026-09-24 03:05:16 on dist-test-slave-wgsb"
server_key: "9721af1ac5b03b5969da67ae425ee356"
server_key_iv: "74f650090565e0613c1136ac3c1bc00f"
server_key_version: "encryptionkey@0"
I20260924 03:05:17.759795 15548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:17.775063 15548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:17.778056 15548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:17.778216 15548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:17.778486 15548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:05:17.778844 15548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:05:17.778893 15548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:17.778950 15548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:05:17.778980 15548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:17.786558 15548 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35215
I20260924 03:05:17.786598 15676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35215 every 8 connection(s)
I20260924 03:05:17.786939 15548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:05:17.788069 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:17.787660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.3:54663)
0924 03:05:17.787849 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:17.787916 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":125,"threads_started":1}
W20260924 03:05:17.788446 15677 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39953 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39953: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
I20260924 03:05:17.788715 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:17.788621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.3:36517)
0924 03:05:17.788669 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:17.788693 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 03:05:17.789023 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:17.788905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.3:60303)
0924 03:05:17.788980 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:17.789003 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 03:05:17.789126 15677 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 03:05:17.789707 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 15548
I20260924 03:05:18.600355 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:18.600093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.1:33911)
0924 03:05:18.600248 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:18.600290 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":66,"threads_started":1}
I20260924 03:05:18.652475 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:18.652249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.2:56369)
0924 03:05:18.652378 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:18.652432 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260924 03:05:18.789641 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:18.789429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.3:46257)
0924 03:05:18.789555 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:18.789589 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
I20260924 03:05:19.045372 15684 raft_consensus.cc:482] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:05:19.045524 15684 raft_consensus.cc:504] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } }
I20260924 03:05:19.045899 15684 leader_election.cc:288] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be77a11fa86949e0a5c7cdf9af71f901 (127.0.98.3:35215), 3d5e3a3c37cd4aeb8c39f300acc4d502 (127.0.98.2:34387)
Sep 24 03:05:19 dist-test-slave-wgsb krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 24 03:05:19 dist-test-slave-wgsb krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260924 03:05:19.056910 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:19.046166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39469 (local address 127.0.98.3:35215)
0924 03:05:19.046444 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:19.046451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:19.046480 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:05:19.046994 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:19.046998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:19.047056 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:19.047207 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:19.048120 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:19.049055 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:19.049976 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:19.050608 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:19.050719 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:19.053383 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:19.053386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:19.053387 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:19.053443 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:19.055240 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:19.055630 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:19.055633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:19.055633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:19.055738 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:19.055921 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:19.055921 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:19.055921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:19.056216 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:19.056303 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:19.056661 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 03:05:19.056795 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":68,"threads_started":1}
I20260924 03:05:19.056910 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:19.046094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35215 (local address 127.0.98.1:39469)
0924 03:05:19.046318 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:19.046366 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:19.046415 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:19.047328 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:19.047329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:19.047760 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:19.048004 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:19.048017 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:19.049154 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:19.049155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:19.049882 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:19.049884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:19.050696 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:19.050697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:19.050736 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:19.051316 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:19.051320 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:19.053288 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:19.055314 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:19.055315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:19.055315 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:19.055528 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:19.055799 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:19.055799 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:19.055799 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:19.055856 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:19.056293 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:19.056301 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:19.056448 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 03:05:19.056580 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 03:05:19.056824 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":172,"thread_start_us":58,"threads_started":1}
I20260924 03:05:19.056957 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:19.046065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34387 (local address 127.0.98.1:50303)
0924 03:05:19.046179 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:19.046192 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:19.046279 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:19.047247 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:19.047249 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:19.047750 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:19.048003 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:19.048014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:19.049073 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:19.049074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:19.049884 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:19.049886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:19.050466 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:19.050467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:19.050597 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:19.051169 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:19.051179 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:19.052794 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:19.055097 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:19.055103 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:19.055105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:19.055366 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:19.055685 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:19.055685 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:19.055686 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:19.055812 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:19.056293 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:19.056301 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:19.056448 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 03:05:19.056580 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 03:05:19.056772 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":4}
I20260924 03:05:19.056957 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:19.046166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50303 (local address 127.0.98.2:34387)
0924 03:05:19.046326 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:19.046338 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:19.046364 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:05:19.047036 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:19.047039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:19.047079 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:19.047207 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:19.048142 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:19.048973 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:19.049977 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:19.050364 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:19.050453 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:19.052941 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:19.052945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:19.052945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:19.052958 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:19.055011 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:19.055454 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:19.055456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:19.055456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:19.055540 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:19.055909 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:19.055910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:19.055910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:19.056216 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:19.056303 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:19.056661 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 03:05:19.056795 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
W20260924 03:05:19.057880 15299 leader_election.cc:341] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d5e3a3c37cd4aeb8c39f300acc4d502 (127.0.98.2:34387): Not found: Tablet not found: 2d35faf1f16149d09aec08ff090c93f0
W20260924 03:05:19.058544 15298 leader_election.cc:341] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be77a11fa86949e0a5c7cdf9af71f901 (127.0.98.3:35215): Not found: Tablet not found: 2d35faf1f16149d09aec08ff090c93f0
I20260924 03:05:19.058605 15298 leader_election.cc:302] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47b995a4829f4aadbf44f7f523331774; no voters: 3d5e3a3c37cd4aeb8c39f300acc4d502, be77a11fa86949e0a5c7cdf9af71f901
I20260924 03:05:19.058782 15684 raft_consensus.cc:2726] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:05:19.601114 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:19.600844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.1:44463)
0924 03:05:19.601019 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:19.601063 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":66,"threads_started":1}
I20260924 03:05:19.653147 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:19.652976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.2:46301)
0924 03:05:19.653077 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:19.653109 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":55,"threads_started":1}
I20260924 03:05:19.790275 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:19.790076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.3:37573)
0924 03:05:19.790200 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:19.790236 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":57,"threads_started":1}
I20260924 03:05:20.601924 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:20.601673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.1:43549)
0924 03:05:20.601829 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:20.601871 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":105,"threads_started":1}
I20260924 03:05:20.653824 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:20.653606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.2:58051)
0924 03:05:20.653745 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:20.653786 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":59,"threads_started":1}
I20260924 03:05:20.790994 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:20.790757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.3:52677)
0924 03:05:20.790915 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:20.790949 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
I20260924 03:05:20.872117 15694 raft_consensus.cc:482] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:05:20.872229 15694 raft_consensus.cc:504] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } }
I20260924 03:05:20.872439 15694 leader_election.cc:288] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be77a11fa86949e0a5c7cdf9af71f901 (127.0.98.3:35215), 3d5e3a3c37cd4aeb8c39f300acc4d502 (127.0.98.2:34387)
W20260924 03:05:20.872956 15298 leader_election.cc:341] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be77a11fa86949e0a5c7cdf9af71f901 (127.0.98.3:35215): Not found: Tablet not found: 2d35faf1f16149d09aec08ff090c93f0
W20260924 03:05:20.873056 15299 leader_election.cc:341] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d5e3a3c37cd4aeb8c39f300acc4d502 (127.0.98.2:34387): Not found: Tablet not found: 2d35faf1f16149d09aec08ff090c93f0
I20260924 03:05:20.873134 15299 leader_election.cc:302] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47b995a4829f4aadbf44f7f523331774; no voters: 3d5e3a3c37cd4aeb8c39f300acc4d502, be77a11fa86949e0a5c7cdf9af71f901
I20260924 03:05:20.873238 15694 raft_consensus.cc:2726] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:05:21.602784 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:21.602510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.1:36015)
0924 03:05:21.602675 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:21.602724 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":71,"threads_started":1}
I20260924 03:05:21.654471 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:21.654260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.2:58991)
0924 03:05:21.654398 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:21.654433 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260924 03:05:21.791719 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:21.791456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.3:35797)
0924 03:05:21.791584 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:21.791679 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260924 03:05:22.603706 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:22.603412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.1:58831)
0924 03:05:22.603564 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:22.603643 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":102,"threads_started":1}
I20260924 03:05:22.655186 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:22.654976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.2:44799)
0924 03:05:22.655103 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:22.655144 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20260924 03:05:22.792482 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:22.792163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.3:34449)
0924 03:05:22.792320 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:22.792419 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":81,"threads_started":1}
I20260924 03:05:23.604454 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:23.604214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.1:37395)
0924 03:05:23.604360 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:23.604403 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":62,"threads_started":1}
I20260924 03:05:23.655982 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:23.655763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.2:60685)
0924 03:05:23.655898 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:23.655935 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
I20260924 03:05:23.793166 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:23.792938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.3:41275)
0924 03:05:23.793086 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:23.793124 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":79,"threads_started":1}
I20260924 03:05:23.846787 15704 raft_consensus.cc:482] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:05:23.846927 15704 raft_consensus.cc:504] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } }
I20260924 03:05:23.847157 15704 leader_election.cc:288] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be77a11fa86949e0a5c7cdf9af71f901 (127.0.98.3:35215), 3d5e3a3c37cd4aeb8c39f300acc4d502 (127.0.98.2:34387)
W20260924 03:05:23.847622 15298 leader_election.cc:341] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be77a11fa86949e0a5c7cdf9af71f901 (127.0.98.3:35215): Not found: Tablet not found: 2d35faf1f16149d09aec08ff090c93f0
W20260924 03:05:23.847700 15299 leader_election.cc:341] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d5e3a3c37cd4aeb8c39f300acc4d502 (127.0.98.2:34387): Not found: Tablet not found: 2d35faf1f16149d09aec08ff090c93f0
I20260924 03:05:23.847774 15299 leader_election.cc:302] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47b995a4829f4aadbf44f7f523331774; no voters: 3d5e3a3c37cd4aeb8c39f300acc4d502, be77a11fa86949e0a5c7cdf9af71f901
I20260924 03:05:23.847860 15704 raft_consensus.cc:2726] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:05:24.605283 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:24.605050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.1:60153)
0924 03:05:24.605200 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:24.605230 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
I20260924 03:05:24.656777 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:24.656508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.2:39315)
0924 03:05:24.656631 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:24.656673 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260924 03:05:24.793912 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:24.793687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.3:48925)
0924 03:05:24.793831 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:24.793870 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260924 03:05:25.606087 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:25.605822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.1:49493)
0924 03:05:25.605996 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:25.606037 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":91,"threads_started":1}
I20260924 03:05:25.657552 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:25.657338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.2:40471)
0924 03:05:25.657474 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:25.657510 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
I20260924 03:05:25.794762 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:25.794462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.3:42469)
0924 03:05:25.794617 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:25.794658 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":89,"threads_started":1}
I20260924 03:05:26.606953 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:26.606686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.1:41043)
0924 03:05:26.606867 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:26.606906 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":99,"threads_started":1}
I20260924 03:05:26.658237 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:26.658000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.2:46141)
0924 03:05:26.658149 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:26.658192 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":60,"threads_started":1}
I20260924 03:05:26.795426 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:26.795220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.3:55637)
0924 03:05:26.795346 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:26.795382 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":57,"threads_started":1}
I20260924 03:05:27.607905 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:27.607672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.1:56427)
0924 03:05:27.607791 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:27.607821 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260924 03:05:27.659004 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:27.658795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.2:32849)
0924 03:05:27.658919 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:27.658959 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260924 03:05:27.796285 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:27.796034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.3:34411)
0924 03:05:27.796198 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:27.796240 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":94,"threads_started":1}
I20260924 03:05:28.124217 15717 raft_consensus.cc:482] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:05:28.124323 15717 raft_consensus.cc:504] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } }
I20260924 03:05:28.124575 15717 leader_election.cc:288] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be77a11fa86949e0a5c7cdf9af71f901 (127.0.98.3:35215), 3d5e3a3c37cd4aeb8c39f300acc4d502 (127.0.98.2:34387)
W20260924 03:05:28.124867 15298 leader_election.cc:341] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be77a11fa86949e0a5c7cdf9af71f901 (127.0.98.3:35215): Not found: Tablet not found: 2d35faf1f16149d09aec08ff090c93f0
W20260924 03:05:28.124997 15299 leader_election.cc:341] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d5e3a3c37cd4aeb8c39f300acc4d502 (127.0.98.2:34387): Not found: Tablet not found: 2d35faf1f16149d09aec08ff090c93f0
I20260924 03:05:28.125061 15299 leader_election.cc:302] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47b995a4829f4aadbf44f7f523331774; no voters: 3d5e3a3c37cd4aeb8c39f300acc4d502, be77a11fa86949e0a5c7cdf9af71f901
I20260924 03:05:28.125182 15717 raft_consensus.cc:2726] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:05:28.608664 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:28.608444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.1:33457)
0924 03:05:28.608576 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:28.608615 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260924 03:05:28.659725 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:28.659471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.2:52855)
0924 03:05:28.659620 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:28.659662 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260924 03:05:28.797000 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:28.796781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.3:35433)
0924 03:05:28.796914 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:28.796955 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260924 03:05:29.609431 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:29.609215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.1:37407)
0924 03:05:29.609343 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:29.609383 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260924 03:05:29.660471 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:29.660232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.2:33121)
0924 03:05:29.660376 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:29.660414 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260924 03:05:29.797691 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:29.797444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.3:44265)
0924 03:05:29.797601 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:29.797633 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
I20260924 03:05:30.058382 15724 raft_consensus.cc:482] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:05:30.058491 15724 raft_consensus.cc:504] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be77a11fa86949e0a5c7cdf9af71f901" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "3d5e3a3c37cd4aeb8c39f300acc4d502" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34387 } } peers { permanent_uuid: "47b995a4829f4aadbf44f7f523331774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41695 } }
I20260924 03:05:30.058751 15724 leader_election.cc:288] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be77a11fa86949e0a5c7cdf9af71f901 (127.0.98.3:35215), 3d5e3a3c37cd4aeb8c39f300acc4d502 (127.0.98.2:34387)
W20260924 03:05:30.059070 15299 leader_election.cc:341] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d5e3a3c37cd4aeb8c39f300acc4d502 (127.0.98.2:34387): Not found: Tablet not found: 2d35faf1f16149d09aec08ff090c93f0
W20260924 03:05:30.059181 15298 leader_election.cc:341] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be77a11fa86949e0a5c7cdf9af71f901 (127.0.98.3:35215): Not found: Tablet not found: 2d35faf1f16149d09aec08ff090c93f0
I20260924 03:05:30.059266 15298 leader_election.cc:302] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47b995a4829f4aadbf44f7f523331774; no voters: 3d5e3a3c37cd4aeb8c39f300acc4d502, be77a11fa86949e0a5c7cdf9af71f901
I20260924 03:05:30.059360 15724 raft_consensus.cc:2726] T 2d35faf1f16149d09aec08ff090c93f0 P 47b995a4829f4aadbf44f7f523331774 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:05:30.610250 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:30.609991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.1:43881)
0924 03:05:30.610156 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:30.610202 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":84,"threads_started":1}
I20260924 03:05:30.661407 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:30.661160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.2:35369)
0924 03:05:30.661310 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:30.661355 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":67,"threads_started":1}
I20260924 03:05:30.798417 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:30.798196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.3:59255)
0924 03:05:30.798329 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:30.798370 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260924 03:05:31.611099 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:31.610847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.1:41493)
0924 03:05:31.611008 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:31.611051 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
I20260924 03:05:31.662345 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:31.662124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.2:34305)
0924 03:05:31.662256 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:31.662297 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":62,"threads_started":1}
I20260924 03:05:31.799142 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:31.798911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.3:33427)
0924 03:05:31.799050 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:31.799096 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260924 03:05:32.611931 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:32.611711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.1:32823)
0924 03:05:32.611843 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:32.611876 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":59,"threads_started":1}
I20260924 03:05:32.663142 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:32.662930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.2:46453)
0924 03:05:32.663063 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:32.663097 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260924 03:05:32.790292 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 15251
I20260924 03:05:32.795053 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 15414
I20260924 03:05:32.799355 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 15548
2026-09-24T03:05:32Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18577 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:05:32 dist-test-slave-wgsb krb5kdc[15735](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:05:32 dist-test-slave-wgsb krb5kdc[15735](info): set up 2 sockets
Sep 24 03:05:32 dist-test-slave-wgsb krb5kdc[15735](info): commencing operation
krb5kdc: starting...
W20260924 03:05:33.863986 392 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:05:33 dist-test-slave-wgsb krb5kdc[15735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:05:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:05:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:05:33.987283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33847
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:05:34.069314 15748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:34.069525 15748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:34.069548 15748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:34.070647 15748 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 03:05:34.070683 15748 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 03:05:34.070695 15748 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 03:05:34.070708 15748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:05:34.070720 15748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:34.070734 15748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:34.070752 15748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:05:34.070765 15748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:05:34.072220 15748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:34.072405 15748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:34.072566 15748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:34.077950 15756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:34.078078 15754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:34.077975 15753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:34.078009 15748 server_base.cc:1034] running on GCE node
I20260924 03:05:34.078528 15748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:34.078768 15748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:34.079938 15748 hybrid_clock.cc:648] HybridClock initialized: now 1790219134079918 us; error 37 us; skew 500 ppm
Sep 24 03:05:34 dist-test-slave-wgsb krb5kdc[15735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:34.082432 15748 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:05:34.082834 15748 webserver.cc:460] Webserver started at http://127.0.98.62:46529/ using document root <none> and password file <none>
I20260924 03:05:34.083006 15748 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:34.083045 15748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:34.083135 15748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:05:34.084194 15748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "8671af924e834134be7f892873b9033c"
format_stamp: "Formatted at 2026-09-24 03:05:34 on dist-test-slave-wgsb"
server_key: "a5899b22e3f32290fc49cafaee8d97ad"
server_key_iv: "86f8dd7e6838dfb8d218e492886a3ab7"
server_key_version: "encryptionkey@0"
I20260924 03:05:34.084503 15748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "8671af924e834134be7f892873b9033c"
format_stamp: "Formatted at 2026-09-24 03:05:34 on dist-test-slave-wgsb"
server_key: "a5899b22e3f32290fc49cafaee8d97ad"
server_key_iv: "86f8dd7e6838dfb8d218e492886a3ab7"
server_key_version: "encryptionkey@0"
I20260924 03:05:34.085757 15748 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:05:34.086440 15763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:34.086617 15748 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:05:34.086699 15748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "8671af924e834134be7f892873b9033c"
format_stamp: "Formatted at 2026-09-24 03:05:34 on dist-test-slave-wgsb"
server_key: "a5899b22e3f32290fc49cafaee8d97ad"
server_key_iv: "86f8dd7e6838dfb8d218e492886a3ab7"
server_key_version: "encryptionkey@0"
I20260924 03:05:34.086753 15748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:34.098729 15748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:34.101091 15748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:34.101202 15748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:34.105144 15748 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33847
I20260924 03:05:34.105208 15815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33847 every 8 connection(s)
I20260924 03:05:34.105582 15748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:05:34.106519 15816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:05:34.108897 15816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c: Bootstrap starting.
I20260924 03:05:34.109458 15816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c: Neither blocks nor log segments found. Creating new log.
I20260924 03:05:34.109716 15816 log.cc:824] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:05:34.110430 15816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c: No bootstrap required, opened a new log
I20260924 03:05:34.112128 15816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8671af924e834134be7f892873b9033c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33847 } }
I20260924 03:05:34.112272 15816 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:05:34.112294 15816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8671af924e834134be7f892873b9033c, State: Initialized, Role: FOLLOWER
I20260924 03:05:34.112468 15816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8671af924e834134be7f892873b9033c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33847 } }
I20260924 03:05:34.112484 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 15748
I20260924 03:05:34.112528 15816 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:05:34.112573 15816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:05:34.112586 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:05:34.112629 15816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:05:34.112787 392 external_mini_cluster.cc:1428] Setting key 8fa3b108c9d908bad663e0d0c4a7bd87
I20260924 03:05:34.113260 15816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8671af924e834134be7f892873b9033c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33847 } }
I20260924 03:05:34.113390 15816 leader_election.cc:302] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8671af924e834134be7f892873b9033c; no voters:
I20260924 03:05:34.113570 15816 leader_election.cc:288] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:05:34.113771 15821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:05:34.113859 15816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:05:34.113932 15821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [term 1 LEADER]: Becoming Leader. State: Replica: 8671af924e834134be7f892873b9033c, State: Running, Role: LEADER
I20260924 03:05:34.114019 15821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8671af924e834134be7f892873b9033c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33847 } }
I20260924 03:05:34.114441 15821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8671af924e834134be7f892873b9033c. Latest consensus state: current_term: 1 leader_uuid: "8671af924e834134be7f892873b9033c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8671af924e834134be7f892873b9033c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33847 } } }
I20260924 03:05:34.114717 15821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [sys.catalog]: This master's current role is: LEADER
I20260924 03:05:34.114642 15822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8671af924e834134be7f892873b9033c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8671af924e834134be7f892873b9033c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33847 } } }
I20260924 03:05:34.114883 15822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [sys.catalog]: This master's current role is: LEADER
I20260924 03:05:34.115027 15830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:05:34.115625 15830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:05:34.117321 15830 catalog_manager.cc:1347] Generated new cluster ID: 89dc1409c8fd4a919da04f993c610a87
I20260924 03:05:34.117372 15830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:05:34 dist-test-slave-wgsb krb5kdc[15735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:05:34.123803 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:34.113305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60346 (local address 127.0.98.62:33847)
0924 03:05:34.113514 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:34.113523 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:34.113563 (+ 40us) server_negotiation.cc:407] Connection header received
0924 03:05:34.113698 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:34.113706 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:34.113815 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:34.115089 (+ 1274us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:34.115213 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.116521 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.117495 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.117914 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.118033 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:34.120469 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:34.120473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:34.120473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:34.120505 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:34.122224 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:34.122597 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:34.122600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:34.122600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:34.122705 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:34.122919 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:34.122920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:34.122920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:34.123170 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:34.123367 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:34.123435 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 03:05:34.123585 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":89,"threads_started":1}
I20260924 03:05:34.149307 15830 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:05:34.149922 15830 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:05:34.156857 15830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c: Generated new TSK 0
I20260924 03:05:34.157061 15830 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:05:34.187892 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33847
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:05:34.269768 15842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:34.269984 15842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:34.270006 15842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:34.271090 15842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:34.271121 15842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:34.271147 15842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:05:34.272619 15842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:34.272819 15842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:34.272993 15842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:34.278151 15848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:34.278174 15850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:34.278157 15847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:34.278349 15842 server_base.cc:1034] running on GCE node
I20260924 03:05:34.278575 15842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:34.278820 15842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:34.279989 15842 hybrid_clock.cc:648] HybridClock initialized: now 1790219134279975 us; error 44 us; skew 500 ppm
Sep 24 03:05:34 dist-test-slave-wgsb krb5kdc[15735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:34.282480 15842 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:05:34.282912 15842 webserver.cc:460] Webserver started at http://127.0.98.1:34821/ using document root <none> and password file <none>
I20260924 03:05:34.283140 15842 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:34.283218 15842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:34.283370 15842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:05:34.284739 15842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/data/instance:
uuid: "03e6f6053460483fbb02ab415d155335"
format_stamp: "Formatted at 2026-09-24 03:05:34 on dist-test-slave-wgsb"
server_key: "70f65d74e002d4e21c4fadf64da7c046"
server_key_iv: "3690aa39e03a923ace616f23ed8bd8bf"
server_key_version: "encryptionkey@0"
I20260924 03:05:34.285184 15842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/wal/instance:
uuid: "03e6f6053460483fbb02ab415d155335"
format_stamp: "Formatted at 2026-09-24 03:05:34 on dist-test-slave-wgsb"
server_key: "70f65d74e002d4e21c4fadf64da7c046"
server_key_iv: "3690aa39e03a923ace616f23ed8bd8bf"
server_key_version: "encryptionkey@0"
I20260924 03:05:34.286664 15842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:05:34.287459 15857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:34.287658 15842 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:05:34.287842 15842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "03e6f6053460483fbb02ab415d155335"
format_stamp: "Formatted at 2026-09-24 03:05:34 on dist-test-slave-wgsb"
server_key: "70f65d74e002d4e21c4fadf64da7c046"
server_key_iv: "3690aa39e03a923ace616f23ed8bd8bf"
server_key_version: "encryptionkey@0"
I20260924 03:05:34.287928 15842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:34.321187 15842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:34.323927 15842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:34.324102 15842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:34.324385 15842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:05:34.324710 15842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:05:34.324749 15842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:34.324801 15842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:05:34.324822 15842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:34.330461 15842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45587
I20260924 03:05:34.330564 15970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45587 every 8 connection(s)
I20260924 03:05:34.330855 15842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:05:34.333133 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 15842
I20260924 03:05:34.333252 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:05:34.333460 392 external_mini_cluster.cc:1428] Setting key 5adc775eca28fec8366587dc678dea6c
Sep 24 03:05:34 dist-test-slave-wgsb krb5kdc[15735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 03:05:34.342658 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:34.331613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42861 (local address 127.0.98.62:33847)
0924 03:05:34.331666 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:34.331669 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:34.331899 (+ 230us) server_negotiation.cc:407] Connection header received
0924 03:05:34.332382 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:34.332383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:34.332399 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:34.332456 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:34.333676 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.334420 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.336028 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.336407 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.336499 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:34.339152 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:34.339154 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:34.339154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:34.339169 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:34.341073 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:34.341514 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:34.341515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:34.341516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:34.341577 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:34.341915 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:34.341917 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:34.341917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:34.342059 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:34.342141 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:34.342506 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 03:05:34.342588 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:05:34.342887 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:34.331554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.1:42861)
0924 03:05:34.331741 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:34.331776 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:34.332261 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:34.332517 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:34.332521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:34.333261 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:34.333564 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:34.333570 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.334521 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.334522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:34.335931 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:34.335934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.336546 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.336548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:34.336736 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:34.337419 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:34.337428 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:34.339021 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:34.341174 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:34.341181 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:34.341183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:34.341428 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:34.341680 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:34.341681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:34.341682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:34.341836 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:34.342151 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:34.342196 (+ 45us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:34.342358 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 03:05:34.342467 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 03:05:34.342607 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":129,"threads_started":1}
I20260924 03:05:34.343495 15971 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33847
I20260924 03:05:34.343628 15971 heartbeater.cc:464] Registering TS with master...
I20260924 03:05:34.343815 15971 heartbeater.cc:511] Master 127.0.98.62:33847 requested a full tablet report, sending...
I20260924 03:05:34.344396 15780 ts_manager.cc:194] Registered new tserver with Master: 03e6f6053460483fbb02ab415d155335 (127.0.98.1:45587)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 03:05:34.345033 15780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:05:34.362512 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33847
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:05:34.445616 15976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:34.445884 15976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:34.445924 15976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:34.447103 15976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:34.447175 15976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:34.447218 15976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:05:34.448710 15976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:34.448989 15976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:34.449218 15976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:34.454308 15982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:34.454283 15981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:34.454537 15976 server_base.cc:1034] running on GCE node
W20260924 03:05:34.454286 15984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:34.454810 15976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:34.455099 15976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:34.456288 15976 hybrid_clock.cc:648] HybridClock initialized: now 1790219134456262 us; error 32 us; skew 500 ppm
Sep 24 03:05:34 dist-test-slave-wgsb krb5kdc[15735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:34.458863 15976 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:05:34.459331 15976 webserver.cc:460] Webserver started at http://127.0.98.2:34561/ using document root <none> and password file <none>
I20260924 03:05:34.459597 15976 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:34.459707 15976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:34.459856 15976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:05:34.460901 15976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/data/instance:
uuid: "49bde0249b674a0bb78e7fbf77856665"
format_stamp: "Formatted at 2026-09-24 03:05:34 on dist-test-slave-wgsb"
server_key: "881167a33f603d927ae6c3e1a2a44eb5"
server_key_iv: "8e0607128fbf9c3f964c7b1c443a8bc6"
server_key_version: "encryptionkey@0"
I20260924 03:05:34.461275 15976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/wal/instance:
uuid: "49bde0249b674a0bb78e7fbf77856665"
format_stamp: "Formatted at 2026-09-24 03:05:34 on dist-test-slave-wgsb"
server_key: "881167a33f603d927ae6c3e1a2a44eb5"
server_key_iv: "8e0607128fbf9c3f964c7b1c443a8bc6"
server_key_version: "encryptionkey@0"
I20260924 03:05:34.462592 15976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:05:34.463392 15991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:34.463632 15976 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:05:34.463727 15976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "49bde0249b674a0bb78e7fbf77856665"
format_stamp: "Formatted at 2026-09-24 03:05:34 on dist-test-slave-wgsb"
server_key: "881167a33f603d927ae6c3e1a2a44eb5"
server_key_iv: "8e0607128fbf9c3f964c7b1c443a8bc6"
server_key_version: "encryptionkey@0"
I20260924 03:05:34.463809 15976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:34.470665 15976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:34.472572 15976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:34.472738 15976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:34.472963 15976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:05:34.473275 15976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:05:34.473330 15976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:34.473372 15976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:05:34.473400 15976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:34.479272 15976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33663
I20260924 03:05:34.479326 16104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33663 every 8 connection(s)
I20260924 03:05:34.479722 15976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 03:05:34 dist-test-slave-wgsb krb5kdc[15735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:05:34.487869 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 15976
I20260924 03:05:34.487975 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:05:34.488194 392 external_mini_cluster.cc:1428] Setting key a23b4d89154a17b850cce9cb888e649f
I20260924 03:05:34.489235 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:34.480471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47745 (local address 127.0.98.62:33847)
0924 03:05:34.480543 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:34.480545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:34.480708 (+ 163us) server_negotiation.cc:407] Connection header received
0924 03:05:34.481173 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:34.481175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:34.481184 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:34.481239 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:34.482093 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.482579 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.483561 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.483851 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.483910 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:34.485932 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:34.485933 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:34.485934 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:34.485940 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:34.487412 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:34.488057 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:34.488058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:34.488058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:34.488132 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:34.488494 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:34.488496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:34.488496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:34.488675 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:34.488761 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:34.489093 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 03:05:34.489169 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 03:05:34.489568 16107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:34.480429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.2:47745)
0924 03:05:34.480584 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:34.480625 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:34.481093 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:34.481288 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:34.481291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:34.481834 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:34.482019 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:34.482023 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.482665 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.482665 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:34.483484 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:34.483486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.483930 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.483931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:34.484038 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:34.484472 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:34.484480 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:34.485822 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:34.487495 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:34.487501 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:34.487504 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:34.487933 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:34.488244 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:34.488245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:34.488245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:34.488412 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:34.488752 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:34.488759 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:34.488940 (+ 181us) client_negotiation.cc:769] Sending connection context
0924 03:05:34.489068 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 03:05:34.489264 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260924 03:05:34.490134 16105 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33847
I20260924 03:05:34.490231 16105 heartbeater.cc:464] Registering TS with master...
I20260924 03:05:34.490480 16105 heartbeater.cc:511] Master 127.0.98.62:33847 requested a full tablet report, sending...
I20260924 03:05:34.490929 15780 ts_manager.cc:194] Registered new tserver with Master: 49bde0249b674a0bb78e7fbf77856665 (127.0.98.2:33663)
I20260924 03:05:34.491470 15780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47745
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:05:34.514910 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33847
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:05:34.603046 16110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:34.603269 16110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:34.603295 16110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:34.604461 16110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:34.604501 16110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:34.604527 16110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:05:34.605919 16110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:34.606106 16110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:34.606276 16110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:34.611342 16116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:34.611332 16118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:34.611336 16115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:34.611433 16110 server_base.cc:1034] running on GCE node
I20260924 03:05:34.611793 16110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:34.611990 16110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:34.613127 16110 hybrid_clock.cc:648] HybridClock initialized: now 1790219134613108 us; error 36 us; skew 500 ppm
Sep 24 03:05:34 dist-test-slave-wgsb krb5kdc[15735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:34.615839 16110 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:05:34.616384 16110 webserver.cc:460] Webserver started at http://127.0.98.3:45851/ using document root <none> and password file <none>
I20260924 03:05:34.616621 16110 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:34.616665 16110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:34.616808 16110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:05:34.617847 16110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/data/instance:
uuid: "20f9d6a148354c12875a38d6dcc21b59"
format_stamp: "Formatted at 2026-09-24 03:05:34 on dist-test-slave-wgsb"
server_key: "79be96403bf6e77327705a2f82c2f6c6"
server_key_iv: "2429f52eeedfd8ffbe4b84d17ed1b84f"
server_key_version: "encryptionkey@0"
I20260924 03:05:34.618211 16110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/wal/instance:
uuid: "20f9d6a148354c12875a38d6dcc21b59"
format_stamp: "Formatted at 2026-09-24 03:05:34 on dist-test-slave-wgsb"
server_key: "79be96403bf6e77327705a2f82c2f6c6"
server_key_iv: "2429f52eeedfd8ffbe4b84d17ed1b84f"
server_key_version: "encryptionkey@0"
I20260924 03:05:34.619510 16110 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 03:05:34.620426 16125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:34.620608 16110 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:05:34.620700 16110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "20f9d6a148354c12875a38d6dcc21b59"
format_stamp: "Formatted at 2026-09-24 03:05:34 on dist-test-slave-wgsb"
server_key: "79be96403bf6e77327705a2f82c2f6c6"
server_key_iv: "2429f52eeedfd8ffbe4b84d17ed1b84f"
server_key_version: "encryptionkey@0"
I20260924 03:05:34.620772 16110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:34.646394 16110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:34.648355 16110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:34.648468 16110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:34.648676 16110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:05:34.648944 16110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:05:34.648974 16110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:34.648998 16110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:05:34.649009 16110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:34.654829 16110 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46863
I20260924 03:05:34.654882 16238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46863 every 8 connection(s)
I20260924 03:05:34.655194 16110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:05:34.660403 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 16110
I20260924 03:05:34.660476 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:05:34.660657 392 external_mini_cluster.cc:1428] Setting key 5394bc6a11dccd590d5a7005a8e8dcec
Sep 24 03:05:34 dist-test-slave-wgsb krb5kdc[15735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:05:34.664876 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:34.655959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47931 (local address 127.0.98.62:33847)
0924 03:05:34.656048 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:34.656051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:34.656235 (+ 184us) server_negotiation.cc:407] Connection header received
0924 03:05:34.656712 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:34.656714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:34.656727 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:34.656795 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:34.657823 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.658309 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.659328 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.659516 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.659610 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:34.662066 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:34.662067 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:34.662068 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:34.662075 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:34.663473 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:34.663913 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:34.663914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:34.663915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:34.663971 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:34.664235 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:34.664236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:34.664237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:34.664360 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:34.664430 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:34.664704 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 03:05:34.664780 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 03:05:34.665084 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:34.655924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.3:47931)
0924 03:05:34.656105 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:34.656143 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:34.656614 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:34.656870 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:34.656874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:34.657530 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:34.657730 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:34.657735 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.658388 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.658389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:34.659186 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:34.659188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.659635 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.659636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:34.659753 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:34.660209 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:34.660217 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:34.661970 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:34.663576 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:34.663582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:34.663584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:34.663833 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:34.664048 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:34.664049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:34.664050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:34.664162 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:34.664433 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:34.664440 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:34.664573 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 03:05:34.664682 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 03:05:34.664848 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":111,"threads_started":1}
I20260924 03:05:34.665573 16239 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33847
I20260924 03:05:34.665688 16239 heartbeater.cc:464] Registering TS with master...
I20260924 03:05:34.665860 16239 heartbeater.cc:511] Master 127.0.98.62:33847 requested a full tablet report, sending...
I20260924 03:05:34.666277 15780 ts_manager.cc:194] Registered new tserver with Master: 20f9d6a148354c12875a38d6dcc21b59 (127.0.98.3:46863)
I20260924 03:05:34.666736 15780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47931
I20260924 03:05:34.672577 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:05:34.680469 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:34.673896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60354 (local address 127.0.98.62:33847)
0924 03:05:34.673976 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:34.673979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:34.674066 (+ 87us) server_negotiation.cc:407] Connection header received
0924 03:05:34.674160 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:34.674161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:34.674172 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:34.674227 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:34.674808 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.675279 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.676111 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.676310 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.676392 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:34.677618 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:34.677619 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:34.677620 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:34.677626 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:34.678998 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:34.679406 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:34.679408 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:34.679408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:34.679498 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:34.679766 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:34.679767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:34.679768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:34.679920 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:34.680031 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:34.680258 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 03:05:34.680338 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 03:05:34.682289 15780 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:05:34.684257 15780 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:05:34.690984 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:34.686847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36740 (local address 127.0.98.3:46863)
0924 03:05:34.687124 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:34.687129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:34.687138 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:05:34.687174 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:34.687176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:34.687193 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:34.687268 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:05:34.687785 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.688804 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.689898 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.690589 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.690666 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:34.690741 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:34.690861 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 03:05:34.690899 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":86,"threads_started":1}
I20260924 03:05:34.691051 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:34.686616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45587 (local address 127.0.0.1:49246)
0924 03:05:34.686865 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:34.686875 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:34.686922 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:34.687259 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:34.687260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:34.687263 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:05:34.687346 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:34.687349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.688379 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.688380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:34.689754 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:34.689757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.690689 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.690690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:34.690750 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:34.690764 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 03:05:34.690823 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 03:05:34.690954 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":106,"thread_start_us":152,"threads_started":1}
I20260924 03:05:34.691090 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:34.686920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49246 (local address 127.0.98.1:45587)
0924 03:05:34.687082 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:34.687086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:34.687095 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:05:34.687131 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:34.687133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:34.687147 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:34.687364 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:05:34.687438 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.688251 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.689890 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.690596 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.690872 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:34.690937 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:34.690963 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 03:05:34.691006 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260924 03:05:34.691424 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:34.686697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33098 (local address 127.0.98.2:33663)
0924 03:05:34.687018 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:34.687024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:34.687035 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:05:34.687075 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:34.687077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:34.687093 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:34.687203 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:05:34.687431 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.688598 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.689841 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.690606 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.691207 (+ 601us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:34.691278 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:34.691304 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 03:05:34.691347 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":87,"threads_started":1}
I20260924 03:05:34.691051 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:34.686591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33663 (local address 127.0.0.1:33098)
0924 03:05:34.686761 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:34.686801 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:34.686879 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:34.687240 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:34.687244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:34.687246 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:05:34.687339 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:34.687341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.688704 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.688705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:34.689765 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:34.689767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.690698 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.690699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:34.690781 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:34.690786 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:05:34.690815 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 03:05:34.690891 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":6,"thread_start_us":42,"threads_started":1}
I20260924 03:05:34.691574 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:34.686615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46863 (local address 127.0.0.1:36740)
0924 03:05:34.687017 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:34.687028 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:34.687084 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:34.687620 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:34.687621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:34.687623 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:05:34.687665 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:34.687666 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.688937 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.688937 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:34.689790 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:34.689792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.690664 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.690665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:34.690749 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:34.690764 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 03:05:34.690845 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 03:05:34.690890 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":12,"thread_start_us":86,"threads_started":1}
I20260924 03:05:34.691968 15905 tablet_service.cc:1467] Processing CreateTablet for tablet 2c1056dbcfbc43d299263cf0b62024f0 (DEFAULT_TABLE table=test-table [id=16d773cf7eb34efaa930681d584d072d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:05:34.692303 15905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c1056dbcfbc43d299263cf0b62024f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:05:34.692423 16173 tablet_service.cc:1467] Processing CreateTablet for tablet 2c1056dbcfbc43d299263cf0b62024f0 (DEFAULT_TABLE table=test-table [id=16d773cf7eb34efaa930681d584d072d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:05:34.692728 16173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c1056dbcfbc43d299263cf0b62024f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:05:34.695065 16256 tablet_bootstrap.cc:492] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335: Bootstrap starting.
I20260924 03:05:34.695406 16257 tablet_bootstrap.cc:492] T 2c1056dbcfbc43d299263cf0b62024f0 P 20f9d6a148354c12875a38d6dcc21b59: Bootstrap starting.
I20260924 03:05:34.695667 16039 tablet_service.cc:1467] Processing CreateTablet for tablet 2c1056dbcfbc43d299263cf0b62024f0 (DEFAULT_TABLE table=test-table [id=16d773cf7eb34efaa930681d584d072d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:05:34.695822 16256 tablet_bootstrap.cc:654] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335: Neither blocks nor log segments found. Creating new log.
I20260924 03:05:34.696007 16039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c1056dbcfbc43d299263cf0b62024f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:05:34.696142 16257 tablet_bootstrap.cc:654] T 2c1056dbcfbc43d299263cf0b62024f0 P 20f9d6a148354c12875a38d6dcc21b59: Neither blocks nor log segments found. Creating new log.
I20260924 03:05:34.696415 16256 log.cc:824] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335: Log is configured to *not* fsync() on all Append() calls
I20260924 03:05:34.696703 16257 log.cc:824] T 2c1056dbcfbc43d299263cf0b62024f0 P 20f9d6a148354c12875a38d6dcc21b59: Log is configured to *not* fsync() on all Append() calls
I20260924 03:05:34.697175 16256 tablet_bootstrap.cc:492] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335: No bootstrap required, opened a new log
I20260924 03:05:34.697270 16256 ts_tablet_manager.cc:1397] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:05:34.697450 16257 tablet_bootstrap.cc:492] T 2c1056dbcfbc43d299263cf0b62024f0 P 20f9d6a148354c12875a38d6dcc21b59: No bootstrap required, opened a new log
I20260924 03:05:34.697527 16257 ts_tablet_manager.cc:1397] T 2c1056dbcfbc43d299263cf0b62024f0 P 20f9d6a148354c12875a38d6dcc21b59: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 03:05:34.698776 16256 raft_consensus.cc:348] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } } peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } }
I20260924 03:05:34.698939 16256 raft_consensus.cc:374] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:05:34.699002 16256 raft_consensus.cc:729] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e6f6053460483fbb02ab415d155335, State: Initialized, Role: FOLLOWER
I20260924 03:05:34.699168 16256 consensus_queue.cc:260] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } } peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } }
I20260924 03:05:34.699334 16260 tablet_bootstrap.cc:492] T 2c1056dbcfbc43d299263cf0b62024f0 P 49bde0249b674a0bb78e7fbf77856665: Bootstrap starting.
I20260924 03:05:34.699581 16256 ts_tablet_manager.cc:1428] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:05:34.699672 16257 raft_consensus.cc:348] T 2c1056dbcfbc43d299263cf0b62024f0 P 20f9d6a148354c12875a38d6dcc21b59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } } peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } }
I20260924 03:05:34.699831 15971 heartbeater.cc:503] Master 127.0.98.62:33847 was elected leader, sending a full tablet report...
I20260924 03:05:34.699908 16257 raft_consensus.cc:374] T 2c1056dbcfbc43d299263cf0b62024f0 P 20f9d6a148354c12875a38d6dcc21b59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:05:34.699947 16257 raft_consensus.cc:729] T 2c1056dbcfbc43d299263cf0b62024f0 P 20f9d6a148354c12875a38d6dcc21b59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f9d6a148354c12875a38d6dcc21b59, State: Initialized, Role: FOLLOWER
I20260924 03:05:34.700059 16260 tablet_bootstrap.cc:654] T 2c1056dbcfbc43d299263cf0b62024f0 P 49bde0249b674a0bb78e7fbf77856665: Neither blocks nor log segments found. Creating new log.
I20260924 03:05:34.700134 16257 consensus_queue.cc:260] T 2c1056dbcfbc43d299263cf0b62024f0 P 20f9d6a148354c12875a38d6dcc21b59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } } peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } }
I20260924 03:05:34.700408 16257 ts_tablet_manager.cc:1428] T 2c1056dbcfbc43d299263cf0b62024f0 P 20f9d6a148354c12875a38d6dcc21b59: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 03:05:34.700448 16239 heartbeater.cc:503] Master 127.0.98.62:33847 was elected leader, sending a full tablet report...
I20260924 03:05:34.700650 16260 log.cc:824] T 2c1056dbcfbc43d299263cf0b62024f0 P 49bde0249b674a0bb78e7fbf77856665: Log is configured to *not* fsync() on all Append() calls
I20260924 03:05:34.701541 16260 tablet_bootstrap.cc:492] T 2c1056dbcfbc43d299263cf0b62024f0 P 49bde0249b674a0bb78e7fbf77856665: No bootstrap required, opened a new log
I20260924 03:05:34.701612 16260 ts_tablet_manager.cc:1397] T 2c1056dbcfbc43d299263cf0b62024f0 P 49bde0249b674a0bb78e7fbf77856665: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:05:34.702875 16260 raft_consensus.cc:348] T 2c1056dbcfbc43d299263cf0b62024f0 P 49bde0249b674a0bb78e7fbf77856665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } } peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } }
I20260924 03:05:34.702998 16260 raft_consensus.cc:374] T 2c1056dbcfbc43d299263cf0b62024f0 P 49bde0249b674a0bb78e7fbf77856665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:05:34.703019 16260 raft_consensus.cc:729] T 2c1056dbcfbc43d299263cf0b62024f0 P 49bde0249b674a0bb78e7fbf77856665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bde0249b674a0bb78e7fbf77856665, State: Initialized, Role: FOLLOWER
I20260924 03:05:34.703090 16260 consensus_queue.cc:260] T 2c1056dbcfbc43d299263cf0b62024f0 P 49bde0249b674a0bb78e7fbf77856665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } } peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } }
I20260924 03:05:34.703305 16260 ts_tablet_manager.cc:1428] T 2c1056dbcfbc43d299263cf0b62024f0 P 49bde0249b674a0bb78e7fbf77856665: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:05:34.703389 16105 heartbeater.cc:503] Master 127.0.98.62:33847 was elected leader, sending a full tablet report...
W20260924 03:05:34.730334 16106 tablet.cc:2369] T 2c1056dbcfbc43d299263cf0b62024f0 P 49bde0249b674a0bb78e7fbf77856665: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:05:34.782569 16261 raft_consensus.cc:482] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:05:34.782752 16261 raft_consensus.cc:504] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } } peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } }
I20260924 03:05:34.783082 16261 leader_election.cc:288] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bde0249b674a0bb78e7fbf77856665 (127.0.98.2:33663), 20f9d6a148354c12875a38d6dcc21b59 (127.0.98.3:46863)
I20260924 03:05:34.786032 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:34.783445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48773 (local address 127.0.98.2:33663)
0924 03:05:34.783489 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:34.783491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:34.783644 (+ 153us) server_negotiation.cc:407] Connection header received
0924 03:05:34.783674 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:34.783675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:34.783686 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:34.783738 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:05:34.783931 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.784400 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.785299 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.785681 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.785735 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:34.785766 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:34.785920 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 03:05:34.785963 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 03:05:34.786041 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:34.783328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33663 (local address 127.0.98.1:48773)
0924 03:05:34.783493 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:34.783504 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:34.783582 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:34.783757 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:34.783758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:34.783759 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:05:34.783854 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:34.783856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.784478 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.784479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:34.785206 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:34.785208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.785763 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.785764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:34.785845 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:34.785849 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:05:34.785891 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 03:05:34.785918 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":114,"threads_started":1}
I20260924 03:05:34.786077 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:34.783414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34033 (local address 127.0.98.3:46863)
0924 03:05:34.783503 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:34.783505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:34.783516 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:05:34.783549 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:34.783550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:34.783563 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:34.783663 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:05:34.783833 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.784338 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.785339 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.785688 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.785773 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:34.785783 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:34.785967 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 03:05:34.786007 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:05:34.786044 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:34.783310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46863 (local address 127.0.98.1:34033)
0924 03:05:34.783371 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:34.783386 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:34.783448 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:34.783674 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:34.783675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:34.783676 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:05:34.783753 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:34.783755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.784419 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.784420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:34.785202 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:34.785205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.785793 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.785794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:34.785868 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:34.785870 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:05:34.785944 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 03:05:34.785975 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":12}
I20260924 03:05:34.786628 16193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1056dbcfbc43d299263cf0b62024f0" candidate_uuid: "03e6f6053460483fbb02ab415d155335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f9d6a148354c12875a38d6dcc21b59" is_pre_election: true
I20260924 03:05:34.786629 16059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1056dbcfbc43d299263cf0b62024f0" candidate_uuid: "03e6f6053460483fbb02ab415d155335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bde0249b674a0bb78e7fbf77856665" is_pre_election: true
I20260924 03:05:34.786753 16193 raft_consensus.cc:2454] T 2c1056dbcfbc43d299263cf0b62024f0 P 20f9d6a148354c12875a38d6dcc21b59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03e6f6053460483fbb02ab415d155335 in term 0.
I20260924 03:05:34.786855 16059 raft_consensus.cc:2454] T 2c1056dbcfbc43d299263cf0b62024f0 P 49bde0249b674a0bb78e7fbf77856665 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03e6f6053460483fbb02ab415d155335 in term 0.
I20260924 03:05:34.787000 15861 leader_election.cc:302] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e6f6053460483fbb02ab415d155335, 20f9d6a148354c12875a38d6dcc21b59; no voters:
I20260924 03:05:34.787119 16261 raft_consensus.cc:2781] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:05:34.787166 16261 raft_consensus.cc:482] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:05:34.787195 16261 raft_consensus.cc:3037] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:05:34.787927 16261 raft_consensus.cc:504] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } } peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } }
I20260924 03:05:34.788082 16261 leader_election.cc:288] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 1 election: Requested vote from peers 49bde0249b674a0bb78e7fbf77856665 (127.0.98.2:33663), 20f9d6a148354c12875a38d6dcc21b59 (127.0.98.3:46863)
I20260924 03:05:34.788251 16193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1056dbcfbc43d299263cf0b62024f0" candidate_uuid: "03e6f6053460483fbb02ab415d155335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f9d6a148354c12875a38d6dcc21b59"
I20260924 03:05:34.788247 16059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1056dbcfbc43d299263cf0b62024f0" candidate_uuid: "03e6f6053460483fbb02ab415d155335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bde0249b674a0bb78e7fbf77856665"
I20260924 03:05:34.788338 16193 raft_consensus.cc:3037] T 2c1056dbcfbc43d299263cf0b62024f0 P 20f9d6a148354c12875a38d6dcc21b59 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:05:34.788335 16059 raft_consensus.cc:3037] T 2c1056dbcfbc43d299263cf0b62024f0 P 49bde0249b674a0bb78e7fbf77856665 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:05:34.789019 16193 raft_consensus.cc:2454] T 2c1056dbcfbc43d299263cf0b62024f0 P 20f9d6a148354c12875a38d6dcc21b59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03e6f6053460483fbb02ab415d155335 in term 1.
I20260924 03:05:34.789131 16059 raft_consensus.cc:2454] T 2c1056dbcfbc43d299263cf0b62024f0 P 49bde0249b674a0bb78e7fbf77856665 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03e6f6053460483fbb02ab415d155335 in term 1.
I20260924 03:05:34.789232 15861 leader_election.cc:302] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e6f6053460483fbb02ab415d155335, 20f9d6a148354c12875a38d6dcc21b59; no voters:
I20260924 03:05:34.789352 16261 raft_consensus.cc:2781] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:05:34.789479 16261 raft_consensus.cc:686] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335 [term 1 LEADER]: Becoming Leader. State: Replica: 03e6f6053460483fbb02ab415d155335, State: Running, Role: LEADER
I20260924 03:05:34.789602 16261 consensus_queue.cc:237] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } } peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } }
I20260924 03:05:34.790516 15780 catalog_manager.cc:5495] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03e6f6053460483fbb02ab415d155335 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "03e6f6053460483fbb02ab415d155335" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } health_report { overall_health: UNKNOWN } } }
I20260924 03:05:34.807987 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:34.805473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49260 (local address 127.0.98.1:45587)
0924 03:05:34.805539 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:34.805541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:34.805552 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:05:34.805589 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:34.805590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:34.805610 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:34.805672 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:05:34.805828 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.806313 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.807078 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:34.807269 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:34.807328 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:34.807494 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:05:34.807683 (+ 189us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:05:34.807827 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:34.807850 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 03:05:34.807875 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 03:05:34.809921 16193 raft_consensus.cc:1261] T 2c1056dbcfbc43d299263cf0b62024f0 P 20f9d6a148354c12875a38d6dcc21b59 [term 1 FOLLOWER]: Refusing update from remote peer 03e6f6053460483fbb02ab415d155335: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:05:34.809921 16059 raft_consensus.cc:1261] T 2c1056dbcfbc43d299263cf0b62024f0 P 49bde0249b674a0bb78e7fbf77856665 [term 1 FOLLOWER]: Refusing update from remote peer 03e6f6053460483fbb02ab415d155335: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:05:34.810318 16261 consensus_queue.cc:1035] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:05:34.810484 16266 consensus_queue.cc:1035] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:05:34.812863 16269 mvcc.cc:205] Tried to move back new op lower bound from 7332737576178622464 to 7332737576098721792. Current Snapshot: MvccSnapshot[applied={T|T < 7332737576178622464}]
I20260924 03:05:34.815140 15780 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:05:34.815249 15780 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:05:34.815971 15780 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c: Sending DeleteTablet for 3 replicas of tablet 2c1056dbcfbc43d299263cf0b62024f0
I20260924 03:05:34.816341 16173 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c1056dbcfbc43d299263cf0b62024f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:05:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36740
I20260924 03:05:34.816367 16039 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c1056dbcfbc43d299263cf0b62024f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:05:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33098
I20260924 03:05:34.816571 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 15842
I20260924 03:05:34.817901 16280 tablet_replica.cc:330] T 2c1056dbcfbc43d299263cf0b62024f0 P 49bde0249b674a0bb78e7fbf77856665: stopping tablet replica
I20260924 03:05:34.818017 16280 raft_consensus.cc:2229] T 2c1056dbcfbc43d299263cf0b62024f0 P 49bde0249b674a0bb78e7fbf77856665 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:05:34.818130 16280 raft_consensus.cc:2258] T 2c1056dbcfbc43d299263cf0b62024f0 P 49bde0249b674a0bb78e7fbf77856665 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:05:34.818288 16279 tablet_replica.cc:330] T 2c1056dbcfbc43d299263cf0b62024f0 P 20f9d6a148354c12875a38d6dcc21b59: stopping tablet replica
I20260924 03:05:34.818390 16279 raft_consensus.cc:2229] T 2c1056dbcfbc43d299263cf0b62024f0 P 20f9d6a148354c12875a38d6dcc21b59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:05:34.818493 16279 raft_consensus.cc:2258] T 2c1056dbcfbc43d299263cf0b62024f0 P 20f9d6a148354c12875a38d6dcc21b59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:05:34.818634 16280 ts_tablet_manager.cc:1905] T 2c1056dbcfbc43d299263cf0b62024f0 P 49bde0249b674a0bb78e7fbf77856665: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:05:34.818909 16279 ts_tablet_manager.cc:1905] T 2c1056dbcfbc43d299263cf0b62024f0 P 20f9d6a148354c12875a38d6dcc21b59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:05:34.820307 16280 ts_tablet_manager.cc:1918] T 2c1056dbcfbc43d299263cf0b62024f0 P 49bde0249b674a0bb78e7fbf77856665: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:05:34.820392 16280 log.cc:1196] T 2c1056dbcfbc43d299263cf0b62024f0 P 49bde0249b674a0bb78e7fbf77856665: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/wal/wals/2c1056dbcfbc43d299263cf0b62024f0
I20260924 03:05:34.820639 16280 ts_tablet_manager.cc:1939] T 2c1056dbcfbc43d299263cf0b62024f0 P 49bde0249b674a0bb78e7fbf77856665: Deleting consensus metadata
I20260924 03:05:34.820843 16279 ts_tablet_manager.cc:1918] T 2c1056dbcfbc43d299263cf0b62024f0 P 20f9d6a148354c12875a38d6dcc21b59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:05:34.820917 16279 log.cc:1196] T 2c1056dbcfbc43d299263cf0b62024f0 P 20f9d6a148354c12875a38d6dcc21b59: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/wal/wals/2c1056dbcfbc43d299263cf0b62024f0
I20260924 03:05:34.821161 16279 ts_tablet_manager.cc:1939] T 2c1056dbcfbc43d299263cf0b62024f0 P 20f9d6a148354c12875a38d6dcc21b59: Deleting consensus metadata
I20260924 03:05:34.821542 15767 catalog_manager.cc:4841] TS 20f9d6a148354c12875a38d6dcc21b59 (127.0.98.3:46863): tablet 2c1056dbcfbc43d299263cf0b62024f0 (table test-table [id=16d773cf7eb34efaa930681d584d072d]) successfully deleted
I20260924 03:05:34.821899 15766 catalog_manager.cc:4841] TS 49bde0249b674a0bb78e7fbf77856665 (127.0.98.2:33663): tablet 2c1056dbcfbc43d299263cf0b62024f0 (table test-table [id=16d773cf7eb34efaa930681d584d072d]) successfully deleted
W20260924 03:05:34.823791 15764 connection.cc:535] client connection to 127.0.98.1:45587 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 03:05:34.823890 15764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 03:05:34.824075 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 15976
I20260924 03:05:34.824949 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:34.824818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45587 (local address 127.0.0.1:49262)
0924 03:05:34.824863 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:34.824907 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260924 03:05:34.825088 15764 catalog_manager.cc:4584] TS 03e6f6053460483fbb02ab415d155335 (127.0.98.1:45587): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c1056dbcfbc43d299263cf0b62024f0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45587: connect: Connection refused (error 111)
I20260924 03:05:34.829166 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 16110
I20260924 03:05:34.835449 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 15748
I20260924 03:05:54.841375 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33847
--webserver_interface=127.0.98.62
--webserver_port=46529
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:05:54.923491 16282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:54.923743 16282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:54.923765 16282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:54.924876 16282 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 03:05:54.924912 16282 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 03:05:54.924925 16282 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 03:05:54.924937 16282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:05:54.924950 16282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:54.924962 16282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:54.924974 16282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:05:54.924986 16282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:05:54.926429 16282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:54.926690 16282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:54.926941 16282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:54.936251 16289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:54.936256 16291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:54.936295 16282 server_base.cc:1034] running on GCE node
W20260924 03:05:54.936256 16288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:54.936904 16282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:54.937247 16282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:54.938410 16282 hybrid_clock.cc:648] HybridClock initialized: now 1790219154938380 us; error 43 us; skew 500 ppm
Sep 24 03:05:54 dist-test-slave-wgsb krb5kdc[15735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:54.945895 16282 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:05:54.946448 16282 webserver.cc:460] Webserver started at http://127.0.98.62:46529/ using document root <none> and password file <none>
I20260924 03:05:54.946704 16282 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:54.946765 16282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:54.948462 16282 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:05:54.949272 16298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:54.949491 16282 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:05:54.949581 16282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "8671af924e834134be7f892873b9033c"
format_stamp: "Formatted at 2026-09-24 03:05:34 on dist-test-slave-wgsb"
server_key: "a5899b22e3f32290fc49cafaee8d97ad"
server_key_iv: "86f8dd7e6838dfb8d218e492886a3ab7"
server_key_version: "encryptionkey@0"
I20260924 03:05:54.949990 16282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:54.969118 16282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:54.971484 16282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:54.971805 16282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:54.976195 16282 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33847
I20260924 03:05:54.976277 16350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33847 every 8 connection(s)
I20260924 03:05:54.976655 16282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:05:54.977471 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 16282
I20260924 03:05:54.977802 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45587
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33847
--webserver_port=34821
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:05:54.978956 16351 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:05:54.979800 16351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c: Bootstrap starting.
I20260924 03:05:54.988533 16351 log.cc:824] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:05:54.990622 16351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:05:54.990999 16351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c: Bootstrap complete.
I20260924 03:05:54.993403 16351 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8671af924e834134be7f892873b9033c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33847 } }
I20260924 03:05:54.993645 16351 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8671af924e834134be7f892873b9033c, State: Initialized, Role: FOLLOWER
I20260924 03:05:54.993824 16351 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8671af924e834134be7f892873b9033c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33847 } }
I20260924 03:05:54.993889 16351 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:05:54.993945 16351 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:05:54.993999 16351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:05:54.995064 16351 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8671af924e834134be7f892873b9033c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33847 } }
I20260924 03:05:54.995155 16351 leader_election.cc:302] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8671af924e834134be7f892873b9033c; no voters:
I20260924 03:05:54.995399 16351 leader_election.cc:288] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 03:05:54.995452 16355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [term 2 FOLLOWER]: Leader election won for term 2
I20260924 03:05:54.995574 16355 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [term 2 LEADER]: Becoming Leader. State: Replica: 8671af924e834134be7f892873b9033c, State: Running, Role: LEADER
I20260924 03:05:54.995718 16351 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:05:54.995694 16355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8671af924e834134be7f892873b9033c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33847 } }
I20260924 03:05:54.996166 16356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8671af924e834134be7f892873b9033c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8671af924e834134be7f892873b9033c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33847 } } }
I20260924 03:05:54.996299 16356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [sys.catalog]: This master's current role is: LEADER
I20260924 03:05:54.996186 16357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8671af924e834134be7f892873b9033c. Latest consensus state: current_term: 2 leader_uuid: "8671af924e834134be7f892873b9033c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8671af924e834134be7f892873b9033c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33847 } } }
I20260924 03:05:54.996397 16357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [sys.catalog]: This master's current role is: LEADER
I20260924 03:05:54.997000 16364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:05:54.997795 16364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:05:54.997895 16364 catalog_manager.cc:1259] Loaded cluster ID: 89dc1409c8fd4a919da04f993c610a87
I20260924 03:05:54.997928 16364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:05:54.999051 16364 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:05:54.999253 16364 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c: Loaded TSK: 0
I20260924 03:05:55.013077 16364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c: Generated new TSK 1
I20260924 03:05:55.013192 16364 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 03:05:55.063678 16353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:55.063916 16353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:55.063956 16353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:55.065124 16353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:55.065191 16353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:55.065239 16353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:05:55.066727 16353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45587
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34821
--tserver_master_addrs=127.0.98.62:33847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:55.066996 16353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:55.067220 16353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:55.073238 16378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:55.073302 16379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:55.073410 16381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:55.073661 16353 server_base.cc:1034] running on GCE node
I20260924 03:05:55.073859 16353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:55.074110 16353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:55.075325 16353 hybrid_clock.cc:648] HybridClock initialized: now 1790219155075306 us; error 34 us; skew 500 ppm
Sep 24 03:05:55 dist-test-slave-wgsb krb5kdc[15735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:55.078305 16353 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:05:55.078752 16353 webserver.cc:460] Webserver started at http://127.0.98.1:34821/ using document root <none> and password file <none>
I20260924 03:05:55.079010 16353 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:55.079098 16353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:55.080672 16353 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:05:55.081532 16388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:55.081758 16353 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:05:55.081876 16353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "03e6f6053460483fbb02ab415d155335"
format_stamp: "Formatted at 2026-09-24 03:05:34 on dist-test-slave-wgsb"
server_key: "70f65d74e002d4e21c4fadf64da7c046"
server_key_iv: "3690aa39e03a923ace616f23ed8bd8bf"
server_key_version: "encryptionkey@0"
I20260924 03:05:55.082283 16353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:55.104687 16353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:55.107120 16353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:55.107309 16353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:55.107558 16353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:05:55.108042 16395 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 03:05:55.109951 16353 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 03:05:55.110031 16353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 03:05:55.110090 16353 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 03:05:55.110671 16353 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 03:05:55.110726 16353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 03:05:55.110805 16395 tablet_bootstrap.cc:492] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335: Bootstrap starting.
I20260924 03:05:55.118124 16353 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45587
I20260924 03:05:55.118281 16502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45587 every 8 connection(s)
I20260924 03:05:55.118546 16353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:05:55.122031 16395 log.cc:824] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335: Log is configured to *not* fsync() on all Append() calls
I20260924 03:05:55.123263 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 16353
I20260924 03:05:55.123565 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33663
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33847
--webserver_port=34561
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:05:55 dist-test-slave-wgsb krb5kdc[15735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:05:55.126662 16395 tablet_bootstrap.cc:492] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:05:55.126991 16395 tablet_bootstrap.cc:492] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335: Bootstrap complete.
I20260924 03:05:55.128212 16395 ts_tablet_manager.cc:1397] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.011s
I20260924 03:05:55.129449 16395 raft_consensus.cc:348] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } } peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } }
I20260924 03:05:55.129768 16395 raft_consensus.cc:729] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e6f6053460483fbb02ab415d155335, State: Initialized, Role: FOLLOWER
I20260924 03:05:55.129928 16395 consensus_queue.cc:260] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } } peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } }
I20260924 03:05:55.130218 16395 ts_tablet_manager.cc:1428] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 03:05:55.130806 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:55.119328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52265 (local address 127.0.98.62:33847)
0924 03:05:55.119541 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:55.119548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:55.119617 (+ 69us) server_negotiation.cc:407] Connection header received
0924 03:05:55.120229 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:55.120235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:55.120283 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:55.120418 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:55.121050 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.121937 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.123109 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.123630 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.123754 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:55.126400 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:55.126404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:55.126405 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:55.126416 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:55.128575 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:55.129115 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:55.129119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:55.129119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:55.129252 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:55.129555 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:55.129556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:55.129556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:55.129869 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:55.129934 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:55.130383 (+ 449us) server_negotiation.cc:299] Negotiation successful
0924 03:05:55.130539 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":90,"threads_started":1}
I20260924 03:05:55.131335 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:55.119237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.1:52265)
0924 03:05:55.119417 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:55.119466 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:55.119998 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:55.120444 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:55.120448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:55.120733 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:55.120915 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:55.120920 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.122065 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.122066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:55.123016 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:55.123018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.123862 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.123863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:55.123982 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:55.124527 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:55.124535 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:55.126269 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:55.128664 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:55.128670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:55.128672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:55.129016 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:55.129338 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:55.129339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:55.129339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:55.129475 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:55.129989 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:55.130009 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:55.130171 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 03:05:55.130913 (+ 742us) client_negotiation.cc:240] Negotiation successful
0924 03:05:55.131104 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
I20260924 03:05:55.132467 16503 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33847
I20260924 03:05:55.132575 16503 heartbeater.cc:464] Registering TS with master...
I20260924 03:05:55.132787 16503 heartbeater.cc:511] Master 127.0.98.62:33847 requested a full tablet report, sending...
I20260924 03:05:55.133704 16315 ts_manager.cc:194] Registered new tserver with Master: 03e6f6053460483fbb02ab415d155335 (127.0.98.1:45587)
I20260924 03:05:55.135089 16315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52265
I20260924 03:05:55.136243 16503 heartbeater.cc:503] Master 127.0.98.62:33847 was elected leader, sending a full tablet report...
I20260924 03:05:55.140426 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:55.134446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45587 (local address 127.0.0.1:35566)
0924 03:05:55.135693 (+ 1247us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:55.135735 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:55.135853 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:55.136221 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:55.136225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:55.136227 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:05:55.136298 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:55.136301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.137753 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.137754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:55.139051 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:55.139053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.140107 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.140108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:55.140200 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:55.140218 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 03:05:55.140289 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 03:05:55.140344 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1214,"thread_start_us":84,"threads_started":1}
I20260924 03:05:55.140463 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:55.135403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35566 (local address 127.0.98.1:45587)
0924 03:05:55.135562 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:55.135568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:55.135809 (+ 241us) server_negotiation.cc:407] Connection header received
0924 03:05:55.135968 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:55.135970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:55.136076 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:55.136152 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:05:55.136460 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.137642 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.139143 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.139991 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.140072 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:55.140137 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:55.140331 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 03:05:55.140384 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260924 03:05:55.141798 16437 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c1056dbcfbc43d299263cf0b62024f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:05:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35566
I20260924 03:05:55.142725 16512 tablet_replica.cc:330] stopping tablet replica
I20260924 03:05:55.142874 16512 raft_consensus.cc:2229] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:05:55.142956 16512 raft_consensus.cc:2258] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:05:55.143321 16512 ts_tablet_manager.cc:1905] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:05:55.146303 16512 ts_tablet_manager.cc:1918] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:05:55.146396 16512 log.cc:1196] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/wal/wals/2c1056dbcfbc43d299263cf0b62024f0
I20260924 03:05:55.146651 16512 ts_tablet_manager.cc:1939] T 2c1056dbcfbc43d299263cf0b62024f0 P 03e6f6053460483fbb02ab415d155335: Deleting consensus metadata
I20260924 03:05:55.147311 16299 catalog_manager.cc:4841] TS 03e6f6053460483fbb02ab415d155335 (127.0.98.1:45587): tablet 2c1056dbcfbc43d299263cf0b62024f0 (table test-table [id=16d773cf7eb34efaa930681d584d072d]) successfully deleted
W20260924 03:05:55.218134 16508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:55.218345 16508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:55.218370 16508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:55.219612 16508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:55.219652 16508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:55.219681 16508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:05:55.221225 16508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33663
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34561
--tserver_master_addrs=127.0.98.62:33847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:55.221467 16508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:55.221730 16508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:55.226853 16517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:55.226888 16520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:55.226881 16518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:55.227116 16508 server_base.cc:1034] running on GCE node
I20260924 03:05:55.227393 16508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:55.227641 16508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:55.228826 16508 hybrid_clock.cc:648] HybridClock initialized: now 1790219155228804 us; error 38 us; skew 500 ppm
Sep 24 03:05:55 dist-test-slave-wgsb krb5kdc[15735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:55.231802 16508 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:05:55.232235 16508 webserver.cc:460] Webserver started at http://127.0.98.2:34561/ using document root <none> and password file <none>
I20260924 03:05:55.232501 16508 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:55.232568 16508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:55.234112 16508 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:05:55.234809 16527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:55.235082 16508 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:05:55.235167 16508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "49bde0249b674a0bb78e7fbf77856665"
format_stamp: "Formatted at 2026-09-24 03:05:34 on dist-test-slave-wgsb"
server_key: "881167a33f603d927ae6c3e1a2a44eb5"
server_key_iv: "8e0607128fbf9c3f964c7b1c443a8bc6"
server_key_version: "encryptionkey@0"
I20260924 03:05:55.235544 16508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:55.249814 16508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:55.252149 16508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:55.252310 16508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:55.252535 16508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:05:55.252925 16508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:05:55.252978 16508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:55.253021 16508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:05:55.253047 16508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:55.258985 16508 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33663
I20260924 03:05:55.259058 16640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33663 every 8 connection(s)
I20260924 03:05:55.259425 16508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:05:55.261387 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 16508
I20260924 03:05:55.261745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46863
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33847
--webserver_port=45851
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:05:55 dist-test-slave-wgsb krb5kdc[15735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:05:55.271905 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:55.260440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36839 (local address 127.0.98.62:33847)
0924 03:05:55.260582 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:55.260585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:55.260756 (+ 171us) server_negotiation.cc:407] Connection header received
0924 03:05:55.261400 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:55.261401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:55.261414 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:55.261531 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:55.262924 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.263443 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.265058 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.265269 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.265340 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:55.268109 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:55.268112 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:55.268112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:55.268122 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:55.270183 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:55.270602 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:55.270603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:55.270604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:55.270668 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:55.271009 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:55.271010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:55.271011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:55.271173 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:55.271255 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:55.271620 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 03:05:55.271744 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 03:05:55.271999 16643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:55.260393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.2:36839)
0924 03:05:55.260612 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:55.260658 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:55.261231 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:55.261627 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:55.261631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:55.262500 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:55.262795 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:55.262801 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.263547 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.263548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:55.264964 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:55.264967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.265371 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.265372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:55.265542 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:55.266187 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:55.266199 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:55.268012 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:55.270284 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:55.270290 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:55.270292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:55.270519 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:55.270805 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:55.270806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:55.270806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:55.270925 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:55.271303 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:55.271310 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:55.271443 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 03:05:55.271543 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 03:05:55.271777 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":90,"threads_started":1}
I20260924 03:05:55.272549 16641 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33847
I20260924 03:05:55.272671 16641 heartbeater.cc:464] Registering TS with master...
I20260924 03:05:55.272902 16641 heartbeater.cc:511] Master 127.0.98.62:33847 requested a full tablet report, sending...
I20260924 03:05:55.273488 16315 ts_manager.cc:194] Registered new tserver with Master: 49bde0249b674a0bb78e7fbf77856665 (127.0.98.2:33663)
I20260924 03:05:55.273927 16315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36839
W20260924 03:05:55.380613 16644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:05:55.380892 16644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:05:55.380921 16644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:05:55.384811 16644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:05:55.384907 16644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:05:55.384945 16644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:05:55.387216 16644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45851
--tserver_master_addrs=127.0.98.62:33847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:05:55.387542 16644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:05:55.387876 16644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:05:55.395073 16649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:55.395076 16652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:05:55.395570 16650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:05:55.397809 16644 server_base.cc:1034] running on GCE node
I20260924 03:05:55.398247 16644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:05:55.399158 16644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:05:55.401117 16644 hybrid_clock.cc:648] HybridClock initialized: now 1790219155401062 us; error 72 us; skew 500 ppm
Sep 24 03:05:55 dist-test-slave-wgsb krb5kdc[15735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:05:55.408013 16644 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:05:55.409260 16644 webserver.cc:460] Webserver started at http://127.0.98.3:45851/ using document root <none> and password file <none>
I20260924 03:05:55.409538 16644 fs_manager.cc:359] Metadata directory not provided
I20260924 03:05:55.409596 16644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:05:55.411912 16644 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:05:55.413121 16659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:55.414227 16644 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 03:05:55.414584 16644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "20f9d6a148354c12875a38d6dcc21b59"
format_stamp: "Formatted at 2026-09-24 03:05:34 on dist-test-slave-wgsb"
server_key: "79be96403bf6e77327705a2f82c2f6c6"
server_key_iv: "2429f52eeedfd8ffbe4b84d17ed1b84f"
server_key_version: "encryptionkey@0"
I20260924 03:05:55.415100 16644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:05:55.428185 16644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:05:55.430693 16644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:05:55.430866 16644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:05:55.431169 16644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:05:55.431497 16644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:05:55.431553 16644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:55.431627 16644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:05:55.431658 16644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:55.438166 16644 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46863
I20260924 03:05:55.438235 16772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46863 every 8 connection(s)
I20260924 03:05:55.438619 16644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:05:55.440234 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 16644
Sep 24 03:05:55 dist-test-slave-wgsb krb5kdc[15735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:05:55.447992 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:55.440671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49834 (local address 127.0.98.62:33847)
0924 03:05:55.440770 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:55.440771 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:55.440898 (+ 127us) server_negotiation.cc:407] Connection header received
0924 03:05:55.440984 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:55.440985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:55.440996 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:55.441062 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:55.441675 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.442202 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.442996 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.443276 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.443351 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:55.444850 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:55.444851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:55.444852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:55.444857 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:55.446610 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:55.447007 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:55.447008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:55.447008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:55.447063 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:55.447299 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:55.447300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:55.447300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:55.447431 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:55.447689 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:55.447728 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 03:05:55.447809 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":63,"threads_started":1}
I20260924 03:05:55.448912 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:55.439512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44357 (local address 127.0.98.62:33847)
0924 03:05:55.439586 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:55.439630 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:55.439861 (+ 231us) server_negotiation.cc:407] Connection header received
0924 03:05:55.440550 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:55.440551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:55.440568 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:55.440654 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:55.441748 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.442239 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.443262 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.443470 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.443558 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:55.445760 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:55.445762 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:55.445762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:55.445768 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:55.447314 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:55.447827 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:55.447829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:55.447829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:55.447886 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:55.448260 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:55.448261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:55.448261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:55.448380 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:55.448455 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:55.448744 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 03:05:55.448828 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 03:05:55.449139 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:55.439491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.3:44357)
0924 03:05:55.439703 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:55.439756 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:55.440409 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:55.440858 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:55.440862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:55.441439 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:05:55.441639 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:55.441643 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.442345 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.442346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:55.443165 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:55.443167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.443571 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.443572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:55.443735 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:55.444162 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:05:55.444170 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:05:55.445651 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:05:55.447395 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:55.447402 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:55.447405 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:55.447720 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:55.447997 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:05:55.447998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:05:55.447998 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:05:55.448145 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:05:55.448462 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:05:55.448468 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:05:55.448601 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 03:05:55.448685 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 03:05:55.448831 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":110,"threads_started":1}
I20260924 03:05:55.449575 16773 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33847
I20260924 03:05:55.449689 16773 heartbeater.cc:464] Registering TS with master...
I20260924 03:05:55.449873 16773 heartbeater.cc:511] Master 127.0.98.62:33847 requested a full tablet report, sending...
I20260924 03:05:55.450249 16315 ts_manager.cc:194] Registered new tserver with Master: 20f9d6a148354c12875a38d6dcc21b59 (127.0.98.3:46863)
I20260924 03:05:55.450616 16315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44357
I20260924 03:05:55.459992 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:05:55.463300 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:55.460382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49840 (local address 127.0.98.62:33847)
0924 03:05:55.460464 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:55.460466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:55.460566 (+ 100us) server_negotiation.cc:407] Connection header received
0924 03:05:55.460673 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:55.460674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:55.460729 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:55.460809 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:05:55.460987 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.461480 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.462287 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.462677 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.462799 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:55.462990 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:05:55.463050 (+ 60us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 03:05:55.463217 (+ 167us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49840: token expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 03:05:55.463393 16506 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49840: token expired
W20260924 03:05:55.463439 16244 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:33847)
I20260924 03:05:55.469713 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:55.463704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49856 (local address 127.0.98.62:33847)
0924 03:05:55.463739 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:55.463741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:55.463814 (+ 73us) server_negotiation.cc:407] Connection header received
0924 03:05:55.464010 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:55.464011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:55.464021 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:55.464076 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:05:55.464597 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.465079 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.465744 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.465960 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.466011 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:55.467364 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:05:55.467366 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:05:55.467366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:05:55.467372 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:05:55.468649 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:55.469002 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:55.469003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:55.469003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:55.469053 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:05:55.469262 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:05:55.469263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:05:55.469263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:05:55.469390 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:05:55.469489 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:55.469588 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 03:05:55.469649 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 03:05:55.470975 16315 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:05:55.472832 16315 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:05:55.475032 16437 tablet_service.cc:1467] Processing CreateTablet for tablet b6a0cd8e1dfd4b57ae03870336a8d40c (DEFAULT_TABLE table=test-table [id=53ddfea101d747c0bf3ba39799116082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:05:55.475242 16437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6a0cd8e1dfd4b57ae03870336a8d40c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:05:55.476953 16395 tablet_bootstrap.cc:492] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335: Bootstrap starting.
I20260924 03:05:55.477290 16395 tablet_bootstrap.cc:654] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335: Neither blocks nor log segments found. Creating new log.
I20260924 03:05:55.477932 16395 tablet_bootstrap.cc:492] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335: No bootstrap required, opened a new log
I20260924 03:05:55.477972 16395 ts_tablet_manager.cc:1397] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 03:05:55.478066 16395 raft_consensus.cc:348] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } } peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } }
I20260924 03:05:55.478127 16395 raft_consensus.cc:374] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:05:55.478174 16395 raft_consensus.cc:729] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e6f6053460483fbb02ab415d155335, State: Initialized, Role: FOLLOWER
I20260924 03:05:55.478226 16395 consensus_queue.cc:260] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } } peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } }
I20260924 03:05:55.478334 16395 ts_tablet_manager.cc:1428] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 03:05:55.479477 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:55.474830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33663 (local address 127.0.0.1:34572)
0924 03:05:55.475075 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:55.475083 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:55.475133 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:55.475992 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:55.475993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:55.475995 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:05:55.476042 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:55.476043 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.477341 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.477342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:55.478363 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:55.478366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.479242 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.479243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:55.479312 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:55.479316 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:05:55.479363 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 03:05:55.479406 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":82,"threads_started":1}
I20260924 03:05:55.479527 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:55.474947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34572 (local address 127.0.98.2:33663)
0924 03:05:55.475733 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:55.475739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:55.475750 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:05:55.475799 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:55.475802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:55.475818 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:55.476116 (+ 298us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:05:55.476273 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.477256 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.478435 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.479170 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.479240 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:55.479297 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:55.479397 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 03:05:55.479442 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":80,"threads_started":1}
I20260924 03:05:55.479661 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:55.474733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46863 (local address 127.0.0.1:40462)
0924 03:05:55.474954 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:55.474964 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:55.475002 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:55.475956 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:55.475958 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:55.475960 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:05:55.476008 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:55.476009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.477006 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.477007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:55.478117 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:55.478120 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.479433 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.479434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:55.479492 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:55.479495 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:05:55.479526 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 03:05:55.479572 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204}
I20260924 03:05:55.480204 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:55.474835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40462 (local address 127.0.98.3:46863)
0924 03:05:55.475740 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:55.475745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:55.475757 (+ 12us) server_negotiation.cc:407] Connection header received
0924 03:05:55.475799 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:55.475801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:55.475818 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:55.475958 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:05:55.476077 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.476901 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.478564 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.479338 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.480053 (+ 715us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:55.480096 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:55.480117 (+ 21us) server_negotiation.cc:299] Negotiation successful
0924 03:05:55.480144 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":870,"thread_start_us":102,"threads_started":1}
I20260924 03:05:55.480137 16575 tablet_service.cc:1467] Processing CreateTablet for tablet b6a0cd8e1dfd4b57ae03870336a8d40c (DEFAULT_TABLE table=test-table [id=53ddfea101d747c0bf3ba39799116082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:05:55.480472 16575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6a0cd8e1dfd4b57ae03870336a8d40c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:05:55.480795 16707 tablet_service.cc:1467] Processing CreateTablet for tablet b6a0cd8e1dfd4b57ae03870336a8d40c (DEFAULT_TABLE table=test-table [id=53ddfea101d747c0bf3ba39799116082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:05:55.481061 16707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6a0cd8e1dfd4b57ae03870336a8d40c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:05:55.483330 16783 tablet_bootstrap.cc:492] T b6a0cd8e1dfd4b57ae03870336a8d40c P 49bde0249b674a0bb78e7fbf77856665: Bootstrap starting.
I20260924 03:05:55.483704 16784 tablet_bootstrap.cc:492] T b6a0cd8e1dfd4b57ae03870336a8d40c P 20f9d6a148354c12875a38d6dcc21b59: Bootstrap starting.
I20260924 03:05:55.484042 16783 tablet_bootstrap.cc:654] T b6a0cd8e1dfd4b57ae03870336a8d40c P 49bde0249b674a0bb78e7fbf77856665: Neither blocks nor log segments found. Creating new log.
I20260924 03:05:55.484421 16784 tablet_bootstrap.cc:654] T b6a0cd8e1dfd4b57ae03870336a8d40c P 20f9d6a148354c12875a38d6dcc21b59: Neither blocks nor log segments found. Creating new log.
I20260924 03:05:55.484723 16783 log.cc:824] T b6a0cd8e1dfd4b57ae03870336a8d40c P 49bde0249b674a0bb78e7fbf77856665: Log is configured to *not* fsync() on all Append() calls
I20260924 03:05:55.485065 16784 log.cc:824] T b6a0cd8e1dfd4b57ae03870336a8d40c P 20f9d6a148354c12875a38d6dcc21b59: Log is configured to *not* fsync() on all Append() calls
I20260924 03:05:55.485616 16783 tablet_bootstrap.cc:492] T b6a0cd8e1dfd4b57ae03870336a8d40c P 49bde0249b674a0bb78e7fbf77856665: No bootstrap required, opened a new log
I20260924 03:05:55.485700 16783 ts_tablet_manager.cc:1397] T b6a0cd8e1dfd4b57ae03870336a8d40c P 49bde0249b674a0bb78e7fbf77856665: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 03:05:55.485800 16784 tablet_bootstrap.cc:492] T b6a0cd8e1dfd4b57ae03870336a8d40c P 20f9d6a148354c12875a38d6dcc21b59: No bootstrap required, opened a new log
I20260924 03:05:55.485874 16784 ts_tablet_manager.cc:1397] T b6a0cd8e1dfd4b57ae03870336a8d40c P 20f9d6a148354c12875a38d6dcc21b59: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:05:55.488089 16784 raft_consensus.cc:348] T b6a0cd8e1dfd4b57ae03870336a8d40c P 20f9d6a148354c12875a38d6dcc21b59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } } peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } }
I20260924 03:05:55.488076 16783 raft_consensus.cc:348] T b6a0cd8e1dfd4b57ae03870336a8d40c P 49bde0249b674a0bb78e7fbf77856665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } } peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } }
I20260924 03:05:55.488282 16783 raft_consensus.cc:374] T b6a0cd8e1dfd4b57ae03870336a8d40c P 49bde0249b674a0bb78e7fbf77856665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:05:55.488282 16784 raft_consensus.cc:374] T b6a0cd8e1dfd4b57ae03870336a8d40c P 20f9d6a148354c12875a38d6dcc21b59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:05:55.488338 16783 raft_consensus.cc:729] T b6a0cd8e1dfd4b57ae03870336a8d40c P 49bde0249b674a0bb78e7fbf77856665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bde0249b674a0bb78e7fbf77856665, State: Initialized, Role: FOLLOWER
I20260924 03:05:55.488338 16784 raft_consensus.cc:729] T b6a0cd8e1dfd4b57ae03870336a8d40c P 20f9d6a148354c12875a38d6dcc21b59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f9d6a148354c12875a38d6dcc21b59, State: Initialized, Role: FOLLOWER
I20260924 03:05:55.488480 16783 consensus_queue.cc:260] T b6a0cd8e1dfd4b57ae03870336a8d40c P 49bde0249b674a0bb78e7fbf77856665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } } peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } }
I20260924 03:05:55.488480 16784 consensus_queue.cc:260] T b6a0cd8e1dfd4b57ae03870336a8d40c P 20f9d6a148354c12875a38d6dcc21b59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } } peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } }
I20260924 03:05:55.488807 16783 ts_tablet_manager.cc:1428] T b6a0cd8e1dfd4b57ae03870336a8d40c P 49bde0249b674a0bb78e7fbf77856665: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260924 03:05:55.488807 16784 ts_tablet_manager.cc:1428] T b6a0cd8e1dfd4b57ae03870336a8d40c P 20f9d6a148354c12875a38d6dcc21b59: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:05:55.488822 16773 heartbeater.cc:503] Master 127.0.98.62:33847 was elected leader, sending a full tablet report...
I20260924 03:05:55.488973 16641 heartbeater.cc:503] Master 127.0.98.62:33847 was elected leader, sending a full tablet report...
W20260924 03:05:55.510092 16642 tablet.cc:2369] T b6a0cd8e1dfd4b57ae03870336a8d40c P 49bde0249b674a0bb78e7fbf77856665: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:05:55.593853 16509 raft_consensus.cc:482] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:05:55.594010 16509 raft_consensus.cc:504] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } } peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } }
I20260924 03:05:55.594326 16509 leader_election.cc:288] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20f9d6a148354c12875a38d6dcc21b59 (127.0.98.3:46863), 49bde0249b674a0bb78e7fbf77856665 (127.0.98.2:33663)
I20260924 03:05:55.597884 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:55.594569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34025 (local address 127.0.98.3:46863)
0924 03:05:55.594617 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:55.594619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:55.594654 (+ 35us) server_negotiation.cc:407] Connection header received
0924 03:05:55.594683 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:55.594684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:55.594696 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:55.594752 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:05:55.594925 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.595584 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.597010 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.597520 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.597598 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:55.597609 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:55.597770 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 03:05:55.597810 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 03:05:55.597864 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:55.594490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46863 (local address 127.0.98.1:34025)
0924 03:05:55.594552 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:55.594565 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:55.594614 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:55.594769 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:55.594770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:55.594772 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:05:55.594833 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:55.594835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.595746 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.595746 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:55.596932 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:55.596934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.597616 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.597617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:55.597672 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:55.597677 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:05:55.597751 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 03:05:55.597785 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 03:05:55.598256 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:55.594616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39349 (local address 127.0.98.2:33663)
0924 03:05:55.594661 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:55.594663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:55.594703 (+ 40us) server_negotiation.cc:407] Connection header received
0924 03:05:55.594781 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:55.594782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:55.594800 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:55.594860 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:05:55.595367 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.596072 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.597292 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.597883 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.597954 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:55.597964 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:55.598146 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 03:05:55.598187 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 03:05:55.598366 16727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a0cd8e1dfd4b57ae03870336a8d40c" candidate_uuid: "03e6f6053460483fbb02ab415d155335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f9d6a148354c12875a38d6dcc21b59" is_pre_election: true
I20260924 03:05:55.598553 16727 raft_consensus.cc:2454] T b6a0cd8e1dfd4b57ae03870336a8d40c P 20f9d6a148354c12875a38d6dcc21b59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03e6f6053460483fbb02ab415d155335 in term 0.
I20260924 03:05:55.598524 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:55.594513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33663 (local address 127.0.98.1:39349)
0924 03:05:55.594648 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:55.594658 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:05:55.594715 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:05:55.594901 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:05:55.594901 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:05:55.594903 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:05:55.595279 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:55.595282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.596145 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.596146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:55.597216 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:05:55.597218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.597981 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.597982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:05:55.598047 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:55.598052 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:05:55.598372 (+ 320us) client_negotiation.cc:240] Negotiation successful
0924 03:05:55.598430 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260924 03:05:55.598807 16392 leader_election.cc:302] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e6f6053460483fbb02ab415d155335, 20f9d6a148354c12875a38d6dcc21b59; no voters:
I20260924 03:05:55.598888 16595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a0cd8e1dfd4b57ae03870336a8d40c" candidate_uuid: "03e6f6053460483fbb02ab415d155335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bde0249b674a0bb78e7fbf77856665" is_pre_election: true
I20260924 03:05:55.598940 16509 raft_consensus.cc:2781] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:05:55.598990 16509 raft_consensus.cc:482] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:05:55.599043 16509 raft_consensus.cc:3037] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:05:55.599046 16595 raft_consensus.cc:2454] T b6a0cd8e1dfd4b57ae03870336a8d40c P 49bde0249b674a0bb78e7fbf77856665 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03e6f6053460483fbb02ab415d155335 in term 0.
I20260924 03:05:55.599807 16509 raft_consensus.cc:504] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } } peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } }
I20260924 03:05:55.599959 16509 leader_election.cc:288] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 1 election: Requested vote from peers 20f9d6a148354c12875a38d6dcc21b59 (127.0.98.3:46863), 49bde0249b674a0bb78e7fbf77856665 (127.0.98.2:33663)
I20260924 03:05:55.600109 16727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a0cd8e1dfd4b57ae03870336a8d40c" candidate_uuid: "03e6f6053460483fbb02ab415d155335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f9d6a148354c12875a38d6dcc21b59"
I20260924 03:05:55.600188 16727 raft_consensus.cc:3037] T b6a0cd8e1dfd4b57ae03870336a8d40c P 20f9d6a148354c12875a38d6dcc21b59 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:05:55.600162 16595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a0cd8e1dfd4b57ae03870336a8d40c" candidate_uuid: "03e6f6053460483fbb02ab415d155335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bde0249b674a0bb78e7fbf77856665"
I20260924 03:05:55.600250 16595 raft_consensus.cc:3037] T b6a0cd8e1dfd4b57ae03870336a8d40c P 49bde0249b674a0bb78e7fbf77856665 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:05:55.600880 16727 raft_consensus.cc:2454] T b6a0cd8e1dfd4b57ae03870336a8d40c P 20f9d6a148354c12875a38d6dcc21b59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03e6f6053460483fbb02ab415d155335 in term 1.
I20260924 03:05:55.601032 16595 raft_consensus.cc:2454] T b6a0cd8e1dfd4b57ae03870336a8d40c P 49bde0249b674a0bb78e7fbf77856665 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03e6f6053460483fbb02ab415d155335 in term 1.
I20260924 03:05:55.601099 16392 leader_election.cc:302] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e6f6053460483fbb02ab415d155335, 20f9d6a148354c12875a38d6dcc21b59; no voters:
I20260924 03:05:55.601186 16509 raft_consensus.cc:2781] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:05:55.601338 16509 raft_consensus.cc:686] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 1 LEADER]: Becoming Leader. State: Replica: 03e6f6053460483fbb02ab415d155335, State: Running, Role: LEADER
I20260924 03:05:55.601426 16509 consensus_queue.cc:237] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } } peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } }
I20260924 03:05:55.602203 16314 catalog_manager.cc:5495] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03e6f6053460483fbb02ab415d155335 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "03e6f6053460483fbb02ab415d155335" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } health_report { overall_health: HEALTHY } } }
W20260924 03:05:55.619913 16504 tablet.cc:2369] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:05:55.625963 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:55.623470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35572 (local address 127.0.98.1:45587)
0924 03:05:55.623538 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 03:05:55.623540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:05:55.623565 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:05:55.623677 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:05:55.623678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:05:55.623698 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:05:55.623785 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:05:55.623952 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.624418 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.625160 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:05:55.625368 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:05:55.625447 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:05:55.625628 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:05:55.625726 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:05:55.625799 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:05:55.625843 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 03:05:55.625868 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 03:05:55.627620 16727 raft_consensus.cc:1261] T b6a0cd8e1dfd4b57ae03870336a8d40c P 20f9d6a148354c12875a38d6dcc21b59 [term 1 FOLLOWER]: Refusing update from remote peer 03e6f6053460483fbb02ab415d155335: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:05:55.627591 16595 raft_consensus.cc:1261] T b6a0cd8e1dfd4b57ae03870336a8d40c P 49bde0249b674a0bb78e7fbf77856665 [term 1 FOLLOWER]: Refusing update from remote peer 03e6f6053460483fbb02ab415d155335: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:05:55.627998 16790 consensus_queue.cc:1035] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:05:55.628137 16509 consensus_queue.cc:1035] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:05:55.631217 16792 mvcc.cc:205] Tried to move back new op lower bound from 7332737661447823360 to 7332737661344169984. Current Snapshot: MvccSnapshot[applied={T|T < 7332737661447823360}]
I20260924 03:05:55.631253 16794 mvcc.cc:205] Tried to move back new op lower bound from 7332737661447823360 to 7332737661344169984. Current Snapshot: MvccSnapshot[applied={T|T < 7332737661447823360}]
I20260924 03:05:55.633867 16314 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:05:55.633972 16314 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:05:55.634783 16314 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c: Sending DeleteTablet for 3 replicas of tablet b6a0cd8e1dfd4b57ae03870336a8d40c
I20260924 03:05:55.635155 16707 tablet_service.cc:1514] Processing DeleteTablet for tablet b6a0cd8e1dfd4b57ae03870336a8d40c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:05:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40462
I20260924 03:05:55.635208 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 16353
I20260924 03:05:55.635214 16575 tablet_service.cc:1514] Processing DeleteTablet for tablet b6a0cd8e1dfd4b57ae03870336a8d40c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:05:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34572
I20260924 03:05:55.635432 16802 tablet_replica.cc:330] T b6a0cd8e1dfd4b57ae03870336a8d40c P 49bde0249b674a0bb78e7fbf77856665: stopping tablet replica
I20260924 03:05:55.635525 16802 raft_consensus.cc:2229] T b6a0cd8e1dfd4b57ae03870336a8d40c P 49bde0249b674a0bb78e7fbf77856665 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:05:55.635650 16802 raft_consensus.cc:2258] T b6a0cd8e1dfd4b57ae03870336a8d40c P 49bde0249b674a0bb78e7fbf77856665 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:05:55.636128 16802 ts_tablet_manager.cc:1905] T b6a0cd8e1dfd4b57ae03870336a8d40c P 49bde0249b674a0bb78e7fbf77856665: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:05:55.637712 16802 ts_tablet_manager.cc:1918] T b6a0cd8e1dfd4b57ae03870336a8d40c P 49bde0249b674a0bb78e7fbf77856665: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:05:55.637784 16802 log.cc:1196] T b6a0cd8e1dfd4b57ae03870336a8d40c P 49bde0249b674a0bb78e7fbf77856665: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/wal/wals/b6a0cd8e1dfd4b57ae03870336a8d40c
I20260924 03:05:55.638098 16802 ts_tablet_manager.cc:1939] T b6a0cd8e1dfd4b57ae03870336a8d40c P 49bde0249b674a0bb78e7fbf77856665: Deleting consensus metadata
I20260924 03:05:55.638543 16301 catalog_manager.cc:4841] TS 49bde0249b674a0bb78e7fbf77856665 (127.0.98.2:33663): tablet b6a0cd8e1dfd4b57ae03870336a8d40c (table test-table [id=53ddfea101d747c0bf3ba39799116082]) successfully deleted
W20260924 03:05:55.641631 16299 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45587 (error 108)
I20260924 03:05:55.641798 16801 tablet_replica.cc:330] T b6a0cd8e1dfd4b57ae03870336a8d40c P 20f9d6a148354c12875a38d6dcc21b59: stopping tablet replica
I20260924 03:05:55.641776 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 16508
I20260924 03:05:55.641919 16801 raft_consensus.cc:2229] T b6a0cd8e1dfd4b57ae03870336a8d40c P 20f9d6a148354c12875a38d6dcc21b59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:05:55.642037 16801 raft_consensus.cc:2258] T b6a0cd8e1dfd4b57ae03870336a8d40c P 20f9d6a148354c12875a38d6dcc21b59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:05:55.643098 16801 ts_tablet_manager.cc:1905] T b6a0cd8e1dfd4b57ae03870336a8d40c P 20f9d6a148354c12875a38d6dcc21b59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:05:55.643661 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:05:55.643488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45587 (local address 127.0.0.1:35582)
0924 03:05:55.643550 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 03:05:55.643617 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 03:05:55.643860 16299 catalog_manager.cc:4584] TS 03e6f6053460483fbb02ab415d155335 (127.0.98.1:45587): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b6a0cd8e1dfd4b57ae03870336a8d40c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45587: connect: Connection refused (error 111)
I20260924 03:05:55.644906 16801 ts_tablet_manager.cc:1918] T b6a0cd8e1dfd4b57ae03870336a8d40c P 20f9d6a148354c12875a38d6dcc21b59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:05:55.644982 16801 log.cc:1196] T b6a0cd8e1dfd4b57ae03870336a8d40c P 20f9d6a148354c12875a38d6dcc21b59: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/wal/wals/b6a0cd8e1dfd4b57ae03870336a8d40c
I20260924 03:05:55.645252 16801 ts_tablet_manager.cc:1939] T b6a0cd8e1dfd4b57ae03870336a8d40c P 20f9d6a148354c12875a38d6dcc21b59: Deleting consensus metadata
I20260924 03:05:55.645916 16302 catalog_manager.cc:4841] TS 20f9d6a148354c12875a38d6dcc21b59 (127.0.98.3:46863): tablet b6a0cd8e1dfd4b57ae03870336a8d40c (table test-table [id=53ddfea101d747c0bf3ba39799116082]) successfully deleted
I20260924 03:05:55.647955 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 16644
I20260924 03:05:55.652632 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 16282
I20260924 03:06:15.659359 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33847
--webserver_interface=127.0.98.62
--webserver_port=46529
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:06:15.745703 16804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:15.745950 16804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:15.745977 16804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:15.747231 16804 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 03:06:15.747280 16804 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 03:06:15.747298 16804 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 03:06:15.747319 16804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:06:15.747336 16804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:15.747357 16804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:15.747378 16804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:06:15.747401 16804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:06:15.749110 16804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:15.749341 16804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:15.749559 16804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:06:15.755537 16804 server_base.cc:1034] running on GCE node
W20260924 03:06:15.755579 16810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:15.755658 16809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:15.755532 16812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:15.755987 16804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:15.756256 16804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:15.757417 16804 hybrid_clock.cc:648] HybridClock initialized: now 1790219175757389 us; error 26 us; skew 500 ppm
Sep 24 03:06:15 dist-test-slave-wgsb krb5kdc[15735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:15.767444 16804 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:06:15.768003 16804 webserver.cc:460] Webserver started at http://127.0.98.62:46529/ using document root <none> and password file <none>
I20260924 03:06:15.768261 16804 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:15.768312 16804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:15.769826 16804 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:06:15.770612 16819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:15.770853 16804 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:06:15.771060 16804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "8671af924e834134be7f892873b9033c"
format_stamp: "Formatted at 2026-09-24 03:05:34 on dist-test-slave-wgsb"
server_key: "a5899b22e3f32290fc49cafaee8d97ad"
server_key_iv: "86f8dd7e6838dfb8d218e492886a3ab7"
server_key_version: "encryptionkey@0"
I20260924 03:06:15.771432 16804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:15.781616 16804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:15.783556 16804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:15.783747 16804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:15.787773 16804 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33847
I20260924 03:06:15.787807 16871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33847 every 8 connection(s)
I20260924 03:06:15.788188 16804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:06:15.789685 16872 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:06:15.790328 16872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c: Bootstrap starting.
I20260924 03:06:15.795018 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 16804
I20260924 03:06:15.795383 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45587
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33847
--webserver_port=34821
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:06:15.799430 16872 log.cc:824] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:15.801730 16872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:06:15.802058 16872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c: Bootstrap complete.
I20260924 03:06:15.804347 16872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8671af924e834134be7f892873b9033c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33847 } }
I20260924 03:06:15.804570 16872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8671af924e834134be7f892873b9033c, State: Initialized, Role: FOLLOWER
I20260924 03:06:15.804756 16872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8671af924e834134be7f892873b9033c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33847 } }
I20260924 03:06:15.804832 16872 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:06:15.804873 16872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:06:15.804901 16872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [term 2 FOLLOWER]: Advancing to term 3
I20260924 03:06:15.805938 16872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8671af924e834134be7f892873b9033c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33847 } }
I20260924 03:06:15.806027 16872 leader_election.cc:302] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8671af924e834134be7f892873b9033c; no voters:
I20260924 03:06:15.806208 16872 leader_election.cc:288] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 03:06:15.806270 16876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [term 3 FOLLOWER]: Leader election won for term 3
I20260924 03:06:15.806423 16876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [term 3 LEADER]: Becoming Leader. State: Replica: 8671af924e834134be7f892873b9033c, State: Running, Role: LEADER
I20260924 03:06:15.806491 16872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:06:15.806515 16876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8671af924e834134be7f892873b9033c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33847 } }
I20260924 03:06:15.806783 16876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8671af924e834134be7f892873b9033c. Latest consensus state: current_term: 3 leader_uuid: "8671af924e834134be7f892873b9033c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8671af924e834134be7f892873b9033c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33847 } } }
I20260924 03:06:15.806851 16876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:15.806979 16877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8671af924e834134be7f892873b9033c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8671af924e834134be7f892873b9033c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33847 } } }
I20260924 03:06:15.807034 16877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:15.807125 16881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:06:15.807924 16881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:06:15.808025 16881 catalog_manager.cc:1259] Loaded cluster ID: 89dc1409c8fd4a919da04f993c610a87
I20260924 03:06:15.808058 16881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:06:15.809340 16881 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:06:15.809525 16881 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c: Loaded TSK: 0
I20260924 03:06:15.809576 16881 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8671af924e834134be7f892873b9033c: Loaded TSK: 1
F20260924 03:06:15.809803 16881 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f03745d59ad google::LogMessage::Fail() at ??:0
@ 0x7f03745d888a google::LogMessage::SendToLog() at ??:0
@ 0x7f03745d54de google::LogMessage::Flush() at ??:0
@ 0x7f03745d6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0376d42104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f0376d3fd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f037c52f1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f037c52f48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f037c52f594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f037c4f5c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f037c4ffad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f03764a30d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0376498954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f03772fc6db start_thread at ??:0
@ 0x7f03730d361f clone at ??:0
W20260924 03:06:15.902675 16874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:15.902899 16874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:15.902925 16874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:15.904493 16874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:15.904575 16874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:15.904608 16874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:06:15.906319 16874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45587
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34821
--tserver_master_addrs=127.0.98.62:33847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:15.906599 16874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:15.906847 16874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:15.913338 16913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:15.913269 16914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:15.913515 16916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:15.913565 16874 server_base.cc:1034] running on GCE node
I20260924 03:06:15.914021 16874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:15.914332 16874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:15.915514 16874 hybrid_clock.cc:648] HybridClock initialized: now 1790219175915489 us; error 48 us; skew 500 ppm
Sep 24 03:06:15 dist-test-slave-wgsb krb5kdc[15735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:15.918181 16874 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:06:15.918625 16874 webserver.cc:460] Webserver started at http://127.0.98.1:34821/ using document root <none> and password file <none>
I20260924 03:06:15.918850 16874 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:15.918915 16874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:15.920370 16874 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260924 03:06:15.921172 16923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:15.921330 16874 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:06:15.921423 16874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "03e6f6053460483fbb02ab415d155335"
format_stamp: "Formatted at 2026-09-24 03:05:34 on dist-test-slave-wgsb"
server_key: "70f65d74e002d4e21c4fadf64da7c046"
server_key_iv: "3690aa39e03a923ace616f23ed8bd8bf"
server_key_version: "encryptionkey@0"
I20260924 03:06:15.921805 16874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:15.942073 16874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:15.944506 16874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:15.944707 16874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:15.944957 16874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:06:15.945456 16930 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 03:06:15.947527 16874 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 03:06:15.947661 16874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 03:06:15.947721 16874 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 03:06:15.948436 16874 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 03:06:15.948500 16874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 03:06:15.948567 16930 tablet_bootstrap.cc:492] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335: Bootstrap starting.
I20260924 03:06:15.956588 16874 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45587
I20260924 03:06:15.956763 17037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45587 every 8 connection(s)
I20260924 03:06:15.957049 16874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:06:15.960166 16930 log.cc:824] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:15.961197 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 16874
I20260924 03:06:15.961478 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33663
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33847
--webserver_port=34561
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:06:15.962932 16930 tablet_bootstrap.cc:492] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:06:15.963271 16930 tablet_bootstrap.cc:492] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335: Bootstrap complete.
I20260924 03:06:15.964540 16930 ts_tablet_manager.cc:1397] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260924 03:06:15.966092 16930 raft_consensus.cc:348] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } } peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } }
I20260924 03:06:15.966454 16930 raft_consensus.cc:729] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e6f6053460483fbb02ab415d155335, State: Initialized, Role: FOLLOWER
I20260924 03:06:15.966634 16930 consensus_queue.cc:260] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } } peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } }
I20260924 03:06:15.966923 16930 ts_tablet_manager.cc:1428] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
W20260924 03:06:16.014957 17040 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:06:15.957774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.1:37931)
0924 03:06:15.958030 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:15.958077 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:15.958525 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:16.014721 (+ 56196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":103,"threads_started":1}
W20260924 03:06:16.015182 17038 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33847 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33847: Client connection negotiation failed: client connection to 127.0.98.62:33847: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 03:06:16.015506 17040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:16.015379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.1:37777)
0924 03:06:16.015430 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:16.015476 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 03:06:16.015854 17040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:16.015766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.1:47771)
0924 03:06:16.015815 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:16.015838 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 03:06:16.015980 17038 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:06:16.046121 17042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:16.046337 17042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:16.046363 17042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:16.047497 17042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:16.047533 17042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:16.047565 17042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:06:16.049036 17042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33663
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34561
--tserver_master_addrs=127.0.98.62:33847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:16.049259 17042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:16.049438 17042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:16.055119 17049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:16.055212 17048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:16.055212 17051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:16.055245 17042 server_base.cc:1034] running on GCE node
I20260924 03:06:16.055783 17042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:16.056078 17042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:16.057273 17042 hybrid_clock.cc:648] HybridClock initialized: now 1790219176057244 us; error 46 us; skew 500 ppm
Sep 24 03:06:16 dist-test-slave-wgsb krb5kdc[15735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:16.060030 17042 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:06:16.060571 17042 webserver.cc:460] Webserver started at http://127.0.98.2:34561/ using document root <none> and password file <none>
I20260924 03:06:16.060810 17042 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:16.060876 17042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:16.062322 17042 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:06:16.063030 17058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:16.063213 17042 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:06:16.063308 17042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "49bde0249b674a0bb78e7fbf77856665"
format_stamp: "Formatted at 2026-09-24 03:05:34 on dist-test-slave-wgsb"
server_key: "881167a33f603d927ae6c3e1a2a44eb5"
server_key_iv: "8e0607128fbf9c3f964c7b1c443a8bc6"
server_key_version: "encryptionkey@0"
I20260924 03:06:16.063725 17042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:16.073380 17042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:16.075802 17042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:16.075960 17042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:16.076215 17042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:06:16.076583 17042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:06:16.076633 17042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:16.076704 17042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:06:16.076746 17042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:16.082854 17042 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33663
I20260924 03:06:16.082906 17171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33663 every 8 connection(s)
I20260924 03:06:16.083280 17042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:06:16.084503 17174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:16.084041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.2:57721)
0924 03:06:16.084287 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:16.084360 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":78,"threads_started":1}
W20260924 03:06:16.084941 17172 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33847 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33847: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
I20260924 03:06:16.085264 17174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:16.085170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.2:53357)
0924 03:06:16.085218 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:16.085244 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 03:06:16.085740 17174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:16.085613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.2:34019)
0924 03:06:16.085678 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:16.085719 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 03:06:16.085914 17172 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 03:06:16.086678 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 17042
I20260924 03:06:16.087016 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46863
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33847
--webserver_port=45851
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:06:16.171690 17175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:16.171898 17175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:16.171923 17175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:16.173180 17175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:16.173224 17175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:16.173251 17175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:06:16.174835 17175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45851
--tserver_master_addrs=127.0.98.62:33847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:16.175088 17175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:16.175320 17175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:16.180473 17181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:16.180531 17180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:16.180559 17183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:16.180743 17175 server_base.cc:1034] running on GCE node
I20260924 03:06:16.180958 17175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:16.181247 17175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:16.182462 17175 hybrid_clock.cc:648] HybridClock initialized: now 1790219176182428 us; error 50 us; skew 500 ppm
Sep 24 03:06:16 dist-test-slave-wgsb krb5kdc[15735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:16.186321 17175 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:06:16.186811 17175 webserver.cc:460] Webserver started at http://127.0.98.3:45851/ using document root <none> and password file <none>
I20260924 03:06:16.187047 17175 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:16.187119 17175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:16.188596 17175 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:06:16.189260 17190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:16.189420 17175 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:06:16.189523 17175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "20f9d6a148354c12875a38d6dcc21b59"
format_stamp: "Formatted at 2026-09-24 03:05:34 on dist-test-slave-wgsb"
server_key: "79be96403bf6e77327705a2f82c2f6c6"
server_key_iv: "2429f52eeedfd8ffbe4b84d17ed1b84f"
server_key_version: "encryptionkey@0"
I20260924 03:06:16.189872 17175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:16.199263 17175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:16.201493 17175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:16.201658 17175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:16.201934 17175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:06:16.202358 17175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:06:16.202435 17175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:16.202474 17175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:06:16.202503 17175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:16.208604 17175 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46863
I20260924 03:06:16.208668 17303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46863 every 8 connection(s)
I20260924 03:06:16.209033 17175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:06:16.210391 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:16.209770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.3:33679)
0924 03:06:16.209954 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:16.210038 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":66,"threads_started":1}
W20260924 03:06:16.210656 17304 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33847 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33847: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
I20260924 03:06:16.210987 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:16.210902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.3:42045)
0924 03:06:16.210941 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:16.210969 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 03:06:16.211292 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:16.211223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.3:37427)
0924 03:06:16.211257 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:16.211278 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 03:06:16.211400 17304 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 03:06:16.212150 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 17175
I20260924 03:06:17.016914 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:17.016519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.1:52795)
0924 03:06:17.016776 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:17.016844 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":121,"threads_started":1}
I20260924 03:06:17.086597 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:17.086308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.2:52791)
0924 03:06:17.086491 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:17.086539 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
I20260924 03:06:17.211982 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:17.211776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.3:53391)
0924 03:06:17.211902 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:17.211937 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20260924 03:06:17.326164 17311 raft_consensus.cc:482] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:06:17.326360 17311 raft_consensus.cc:504] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } } peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } }
I20260924 03:06:17.326924 17311 leader_election.cc:288] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20f9d6a148354c12875a38d6dcc21b59 (127.0.98.3:46863), 49bde0249b674a0bb78e7fbf77856665 (127.0.98.2:33663)
Sep 24 03:06:17 dist-test-slave-wgsb krb5kdc[15735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 24 03:06:17 dist-test-slave-wgsb krb5kdc[15735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 03:06:17.340898 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:17.327154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46863 (local address 127.0.98.1:58233)
0924 03:06:17.327234 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:17.327247 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:17.327321 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:17.328563 (+ 1242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:17.328565 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:17.328947 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:06:17.329159 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:17.329165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:17.330921 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:17.330921 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:17.332282 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:17.332285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:17.333204 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:17.333205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:17.333375 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:17.334123 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:06:17.334126 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:06:17.336245 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:06:17.338936 (+ 2691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:06:17.338948 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:06:17.338951 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:06:17.339289 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:06:17.339703 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:06:17.339704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:06:17.339704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:06:17.339861 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:06:17.340357 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:06:17.340364 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:06:17.340497 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 03:06:17.340606 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 03:06:17.340787 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":49}
I20260924 03:06:17.340909 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:17.327288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58233 (local address 127.0.98.3:46863)
0924 03:06:17.327645 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:17.327653 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:17.327681 (+ 28us) server_negotiation.cc:407] Connection header received
0924 03:06:17.328271 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:17.328274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:17.328335 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:17.328495 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:17.329307 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:17.330697 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:17.332385 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:17.333059 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:17.333187 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:17.336427 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:17.336432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:17.336432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:06:17.336445 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:17.338830 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:17.339401 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:17.339404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:17.339405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:17.339533 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:17.339937 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:17.339938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:17.339938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:17.340280 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:17.340378 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:17.340663 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 03:06:17.340787 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":221,"threads_started":1}
I20260924 03:06:17.341569 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:17.327193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33663 (local address 127.0.98.1:38801)
0924 03:06:17.327492 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:17.327510 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:17.327638 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:17.328626 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:17.328627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:17.328973 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:06:17.329159 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:17.329166 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:17.330906 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:17.330908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:17.332282 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:17.332285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:17.333204 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:17.333205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:17.333375 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:17.334109 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:06:17.334120 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:06:17.336546 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:06:17.339851 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:06:17.339852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:06:17.339852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:06:17.340147 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:06:17.340447 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:06:17.340448 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:06:17.340449 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:06:17.340525 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:06:17.340936 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:06:17.340937 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:06:17.341258 (+ 321us) client_negotiation.cc:769] Sending connection context
0924 03:06:17.341331 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 03:06:17.341458 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":82,"thread_start_us":95,"threads_started":1}
I20260924 03:06:17.341688 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:17.327381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38801 (local address 127.0.98.2:33663)
0924 03:06:17.327611 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:17.327627 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:17.327672 (+ 45us) server_negotiation.cc:407] Connection header received
0924 03:06:17.328271 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:17.328275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:17.328337 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:17.328506 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:17.329271 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:17.330730 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:17.332392 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:17.333094 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:17.333187 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:17.336670 (+ 3483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:17.336674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:17.336674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:06:17.336685 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:17.339748 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:17.340255 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:17.340263 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:17.340264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:17.340358 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:17.340602 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:17.340603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:17.340603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:17.340844 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:17.340954 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:17.341413 (+ 459us) server_negotiation.cc:299] Negotiation successful
0924 03:06:17.341554 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":99,"threads_started":1}
W20260924 03:06:17.341826 16927 leader_election.cc:341] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20f9d6a148354c12875a38d6dcc21b59 (127.0.98.3:46863): Not found: Tablet not found: b6a0cd8e1dfd4b57ae03870336a8d40c
W20260924 03:06:17.342344 16926 leader_election.cc:341] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bde0249b674a0bb78e7fbf77856665 (127.0.98.2:33663): Not found: Tablet not found: b6a0cd8e1dfd4b57ae03870336a8d40c
I20260924 03:06:17.342422 16926 leader_election.cc:302] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03e6f6053460483fbb02ab415d155335; no voters: 20f9d6a148354c12875a38d6dcc21b59, 49bde0249b674a0bb78e7fbf77856665
I20260924 03:06:17.342547 17311 raft_consensus.cc:2726] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:06:18.017791 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:18.017581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.1:39847)
0924 03:06:18.017709 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:18.017745 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260924 03:06:18.087478 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:18.087202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.2:40149)
0924 03:06:18.087367 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:18.087417 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":71,"threads_started":1}
I20260924 03:06:18.212720 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:18.212500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.3:56305)
0924 03:06:18.212642 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:18.212678 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":62,"threads_started":1}
I20260924 03:06:19.019254 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:19.018888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.1:46639)
0924 03:06:19.019107 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:19.019168 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":105,"threads_started":1}
I20260924 03:06:19.088222 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:19.088010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.2:57383)
0924 03:06:19.088138 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:19.088173 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260924 03:06:19.213632 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:19.213331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.3:38821)
0924 03:06:19.213531 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:19.213577 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":112,"threads_started":1}
I20260924 03:06:19.357054 17321 raft_consensus.cc:482] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:06:19.357156 17321 raft_consensus.cc:504] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } } peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } }
I20260924 03:06:19.357486 17321 leader_election.cc:288] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20f9d6a148354c12875a38d6dcc21b59 (127.0.98.3:46863), 49bde0249b674a0bb78e7fbf77856665 (127.0.98.2:33663)
W20260924 03:06:19.358000 16926 leader_election.cc:341] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bde0249b674a0bb78e7fbf77856665 (127.0.98.2:33663): Not found: Tablet not found: b6a0cd8e1dfd4b57ae03870336a8d40c
W20260924 03:06:19.358089 16927 leader_election.cc:341] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20f9d6a148354c12875a38d6dcc21b59 (127.0.98.3:46863): Not found: Tablet not found: b6a0cd8e1dfd4b57ae03870336a8d40c
I20260924 03:06:19.358135 16927 leader_election.cc:302] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03e6f6053460483fbb02ab415d155335; no voters: 20f9d6a148354c12875a38d6dcc21b59, 49bde0249b674a0bb78e7fbf77856665
I20260924 03:06:19.358217 17321 raft_consensus.cc:2726] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:06:20.020349 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:20.020084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.1:47503)
0924 03:06:20.020254 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:20.020300 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":74,"threads_started":1}
I20260924 03:06:20.089308 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:20.089037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.2:59587)
0924 03:06:20.089210 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:20.089247 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":96,"threads_started":1}
I20260924 03:06:20.214779 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:20.214323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.3:43841)
0924 03:06:20.214580 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:20.214687 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":129,"threads_started":1}
I20260924 03:06:21.021363 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:21.021112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.1:47383)
0924 03:06:21.021273 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:21.021311 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":100,"threads_started":1}
I20260924 03:06:21.090139 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:21.089852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.2:49307)
0924 03:06:21.090032 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:21.090091 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":67,"threads_started":1}
I20260924 03:06:21.139695 17327 raft_consensus.cc:482] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:06:21.139801 17327 raft_consensus.cc:504] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } } peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } }
I20260924 03:06:21.139995 17327 leader_election.cc:288] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20f9d6a148354c12875a38d6dcc21b59 (127.0.98.3:46863), 49bde0249b674a0bb78e7fbf77856665 (127.0.98.2:33663)
W20260924 03:06:21.140403 16926 leader_election.cc:341] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bde0249b674a0bb78e7fbf77856665 (127.0.98.2:33663): Not found: Tablet not found: b6a0cd8e1dfd4b57ae03870336a8d40c
W20260924 03:06:21.140512 16927 leader_election.cc:341] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20f9d6a148354c12875a38d6dcc21b59 (127.0.98.3:46863): Not found: Tablet not found: b6a0cd8e1dfd4b57ae03870336a8d40c
I20260924 03:06:21.140591 16926 leader_election.cc:302] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03e6f6053460483fbb02ab415d155335; no voters: 20f9d6a148354c12875a38d6dcc21b59, 49bde0249b674a0bb78e7fbf77856665
I20260924 03:06:21.140671 17327 raft_consensus.cc:2726] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:06:21.216143 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:21.215911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.3:48389)
0924 03:06:21.216051 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:21.216093 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
I20260924 03:06:22.022145 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:22.021861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.1:33113)
0924 03:06:22.022045 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:22.022094 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
I20260924 03:06:22.090929 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:22.090669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.2:56691)
0924 03:06:22.090845 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:22.090883 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":62,"threads_started":1}
I20260924 03:06:22.216893 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:22.216656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.3:52521)
0924 03:06:22.216806 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:22.216847 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
I20260924 03:06:23.023031 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:23.022746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.1:52771)
0924 03:06:23.022933 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:23.022982 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":84,"threads_started":1}
I20260924 03:06:23.091845 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:23.091525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.2:60639)
0924 03:06:23.091751 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:23.091795 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":147,"threads_started":1}
I20260924 03:06:23.217818 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:23.217588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.3:35809)
0924 03:06:23.217722 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:23.217768 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
I20260924 03:06:24.023948 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:24.023660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.1:43201)
0924 03:06:24.023848 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:24.023896 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":83,"threads_started":1}
I20260924 03:06:24.092773 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:24.092455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.2:52073)
0924 03:06:24.092666 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:24.092723 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":67,"threads_started":1}
I20260924 03:06:24.218528 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:24.218328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.3:58345)
0924 03:06:24.218450 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:24.218482 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260924 03:06:24.223517 17338 raft_consensus.cc:482] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:06:24.223654 17338 raft_consensus.cc:504] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } } peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } }
I20260924 03:06:24.223898 17338 leader_election.cc:288] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20f9d6a148354c12875a38d6dcc21b59 (127.0.98.3:46863), 49bde0249b674a0bb78e7fbf77856665 (127.0.98.2:33663)
W20260924 03:06:24.224241 16927 leader_election.cc:341] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20f9d6a148354c12875a38d6dcc21b59 (127.0.98.3:46863): Not found: Tablet not found: b6a0cd8e1dfd4b57ae03870336a8d40c
W20260924 03:06:24.224304 16926 leader_election.cc:341] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bde0249b674a0bb78e7fbf77856665 (127.0.98.2:33663): Not found: Tablet not found: b6a0cd8e1dfd4b57ae03870336a8d40c
I20260924 03:06:24.224334 16927 leader_election.cc:302] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03e6f6053460483fbb02ab415d155335; no voters: 20f9d6a148354c12875a38d6dcc21b59, 49bde0249b674a0bb78e7fbf77856665
I20260924 03:06:24.224411 17338 raft_consensus.cc:2726] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:06:25.024871 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:25.024423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.1:46139)
0924 03:06:25.024657 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:25.024736 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":82,"threads_started":1}
I20260924 03:06:25.094334 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:25.094073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.2:39391)
0924 03:06:25.094236 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:25.094285 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":73,"threads_started":1}
I20260924 03:06:25.219341 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:25.219049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.3:56715)
0924 03:06:25.219251 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:25.219292 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":77,"threads_started":1}
I20260924 03:06:25.755348 17342 raft_consensus.cc:482] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:06:25.755451 17342 raft_consensus.cc:504] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9d6a148354c12875a38d6dcc21b59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46863 } } peers { permanent_uuid: "49bde0249b674a0bb78e7fbf77856665" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33663 } } peers { permanent_uuid: "03e6f6053460483fbb02ab415d155335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45587 } }
I20260924 03:06:25.755757 17342 leader_election.cc:288] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20f9d6a148354c12875a38d6dcc21b59 (127.0.98.3:46863), 49bde0249b674a0bb78e7fbf77856665 (127.0.98.2:33663)
W20260924 03:06:25.756130 16927 leader_election.cc:341] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20f9d6a148354c12875a38d6dcc21b59 (127.0.98.3:46863): Not found: Tablet not found: b6a0cd8e1dfd4b57ae03870336a8d40c
W20260924 03:06:25.756209 16926 leader_election.cc:341] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bde0249b674a0bb78e7fbf77856665 (127.0.98.2:33663): Not found: Tablet not found: b6a0cd8e1dfd4b57ae03870336a8d40c
I20260924 03:06:25.756268 16926 leader_election.cc:302] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03e6f6053460483fbb02ab415d155335; no voters: 20f9d6a148354c12875a38d6dcc21b59, 49bde0249b674a0bb78e7fbf77856665
I20260924 03:06:25.756359 17342 raft_consensus.cc:2726] T b6a0cd8e1dfd4b57ae03870336a8d40c P 03e6f6053460483fbb02ab415d155335 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:06:26.025852 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:26.025572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.1:40377)
0924 03:06:26.025754 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:26.025802 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":71,"threads_started":1}
I20260924 03:06:26.095163 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:26.094931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.2:49265)
0924 03:06:26.095077 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:26.095116 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":62,"threads_started":1}
I20260924 03:06:26.220144 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:26.219907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.3:45883)
0924 03:06:26.220061 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:26.220099 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
I20260924 03:06:27.026810 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:27.026492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.1:37591)
0924 03:06:27.026714 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:27.026762 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":106,"threads_started":1}
I20260924 03:06:27.096310 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:27.096007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.2:35359)
0924 03:06:27.096191 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:27.096244 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":71,"threads_started":1}
I20260924 03:06:27.221187 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:27.220833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.3:43125)
0924 03:06:27.221087 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:27.221132 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":88,"threads_started":1}
I20260924 03:06:28.027812 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:28.027456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.1:49935)
0924 03:06:28.027722 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:28.027764 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":77,"threads_started":1}
I20260924 03:06:28.097344 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:28.097058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.2:37319)
0924 03:06:28.097230 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:28.097294 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":65,"threads_started":1}
I20260924 03:06:28.222031 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:28.221794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.3:43423)
0924 03:06:28.221943 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:28.221984 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
I20260924 03:06:29.028698 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:29.028396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.1:52447)
0924 03:06:29.028599 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:29.028648 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":87,"threads_started":1}
I20260924 03:06:29.098143 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:29.097897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.2:48781)
0924 03:06:29.098056 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:29.098094 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":115,"threads_started":1}
I20260924 03:06:29.222877 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:29.222596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.3:34993)
0924 03:06:29.222775 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:29.222819 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":103,"threads_started":1}
I20260924 03:06:30.029451 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:30.029217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.1:50865)
0924 03:06:30.029342 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:30.029383 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
I20260924 03:06:30.098902 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:30.098666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.2:35301)
0924 03:06:30.098809 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:30.098854 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
I20260924 03:06:30.223733 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:30.223433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.3:41811)
0924 03:06:30.223578 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:30.223683 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":63,"threads_started":1}
I20260924 03:06:31.030320 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:31.030091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.1:37087)
0924 03:06:31.030234 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:31.030274 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
I20260924 03:06:31.099768 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:31.099511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.2:54919)
0924 03:06:31.099688 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:31.099723 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":67,"threads_started":1}
I20260924 03:06:31.215970 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 16874
I20260924 03:06:31.221009 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 17042
I20260924 03:06:31.225535 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:31.225311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33847 (local address 127.0.98.3:35969)
0924 03:06:31.225456 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:31.225489 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260924 03:06:31.226310 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 17175
2026-09-24T03:06:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58427 ms)
[----------] 30 tests from SecurityITest (242947 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:06:31 dist-test-slave-wgsb krb5kdc[17363](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:06:31 dist-test-slave-wgsb krb5kdc[17363](info): set up 2 sockets
Sep 24 03:06:31 dist-test-slave-wgsb krb5kdc[17363](info): commencing operation
krb5kdc: starting...
W20260924 03:06:32.339421 392 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:06:32 dist-test-slave-wgsb krb5kdc[17363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:06:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:06:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:06:32.467700 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33653
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:06:32.552670 17376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:32.552922 17376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:32.552960 17376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:32.554139 17376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:06:32.554212 17376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:32.554239 17376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:32.554271 17376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:06:32.554292 17376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:06:32.555840 17376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:32.556108 17376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:32.556335 17376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:06:32.562079 17376 server_base.cc:1034] running on GCE node
W20260924 03:06:32.562079 17382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:32.562209 17381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:32.562240 17384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:32.562613 17376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:32.562898 17376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:32.564078 17376 hybrid_clock.cc:648] HybridClock initialized: now 1790219192564064 us; error 30 us; skew 500 ppm
Sep 24 03:06:32 dist-test-slave-wgsb krb5kdc[17363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:32.566751 17376 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:06:32.567263 17376 webserver.cc:460] Webserver started at http://127.0.98.62:41631/ using document root <none> and password file <none>
I20260924 03:06:32.567502 17376 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:32.567569 17376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:32.567734 17376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:32.568853 17376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "ccdab060b5bc4248af1ae024d28208f8"
format_stamp: "Formatted at 2026-09-24 03:06:32 on dist-test-slave-wgsb"
server_key: "87c33105ed1979872a19db9e5adb16bb"
server_key_iv: "3ecfb4d69252d6c0b8b9f09c9f3eeef9"
server_key_version: "encryptionkey@0"
I20260924 03:06:32.569231 17376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "ccdab060b5bc4248af1ae024d28208f8"
format_stamp: "Formatted at 2026-09-24 03:06:32 on dist-test-slave-wgsb"
server_key: "87c33105ed1979872a19db9e5adb16bb"
server_key_iv: "3ecfb4d69252d6c0b8b9f09c9f3eeef9"
server_key_version: "encryptionkey@0"
I20260924 03:06:32.570675 17376 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:06:32.571461 17391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:32.571700 17376 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:06:32.571801 17376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "ccdab060b5bc4248af1ae024d28208f8"
format_stamp: "Formatted at 2026-09-24 03:06:32 on dist-test-slave-wgsb"
server_key: "87c33105ed1979872a19db9e5adb16bb"
server_key_iv: "3ecfb4d69252d6c0b8b9f09c9f3eeef9"
server_key_version: "encryptionkey@0"
I20260924 03:06:32.571882 17376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:32.585049 17376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:32.588126 17376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:32.588501 17376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:32.594085 17376 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33653
I20260924 03:06:32.594168 17443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33653 every 8 connection(s)
I20260924 03:06:32.594519 17376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:06:32.595392 17444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:32.597802 17444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccdab060b5bc4248af1ae024d28208f8: Bootstrap starting.
I20260924 03:06:32.598414 17444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccdab060b5bc4248af1ae024d28208f8: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:32.598719 17444 log.cc:824] T 00000000000000000000000000000000 P ccdab060b5bc4248af1ae024d28208f8: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:32.600127 17444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccdab060b5bc4248af1ae024d28208f8: No bootstrap required, opened a new log
I20260924 03:06:32.601773 17444 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccdab060b5bc4248af1ae024d28208f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccdab060b5bc4248af1ae024d28208f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33653 } }
I20260924 03:06:32.601917 17444 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccdab060b5bc4248af1ae024d28208f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:32.601938 17444 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccdab060b5bc4248af1ae024d28208f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccdab060b5bc4248af1ae024d28208f8, State: Initialized, Role: FOLLOWER
I20260924 03:06:32.602056 17444 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccdab060b5bc4248af1ae024d28208f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccdab060b5bc4248af1ae024d28208f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33653 } }
I20260924 03:06:32.602149 17444 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccdab060b5bc4248af1ae024d28208f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:06:32.602177 17444 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccdab060b5bc4248af1ae024d28208f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:06:32.602221 17444 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccdab060b5bc4248af1ae024d28208f8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:32.602787 17444 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccdab060b5bc4248af1ae024d28208f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccdab060b5bc4248af1ae024d28208f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33653 } }
I20260924 03:06:32.602871 17444 leader_election.cc:302] T 00000000000000000000000000000000 P ccdab060b5bc4248af1ae024d28208f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccdab060b5bc4248af1ae024d28208f8; no voters:
I20260924 03:06:32.603082 17444 leader_election.cc:288] T 00000000000000000000000000000000 P ccdab060b5bc4248af1ae024d28208f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:06:32.603122 17447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccdab060b5bc4248af1ae024d28208f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:06:32.603248 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 17376
I20260924 03:06:32.603354 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:06:32.603384 17444 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccdab060b5bc4248af1ae024d28208f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:06:32.603394 17447 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccdab060b5bc4248af1ae024d28208f8 [term 1 LEADER]: Becoming Leader. State: Replica: ccdab060b5bc4248af1ae024d28208f8, State: Running, Role: LEADER
I20260924 03:06:32.603561 392 external_mini_cluster.cc:1428] Setting key ade91b2fc73353ad0033f1b470f13c91
I20260924 03:06:32.603675 17447 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccdab060b5bc4248af1ae024d28208f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccdab060b5bc4248af1ae024d28208f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33653 } }
I20260924 03:06:32.604236 17449 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccdab060b5bc4248af1ae024d28208f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccdab060b5bc4248af1ae024d28208f8. Latest consensus state: current_term: 1 leader_uuid: "ccdab060b5bc4248af1ae024d28208f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccdab060b5bc4248af1ae024d28208f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33653 } } }
I20260924 03:06:32.604257 17448 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccdab060b5bc4248af1ae024d28208f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccdab060b5bc4248af1ae024d28208f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccdab060b5bc4248af1ae024d28208f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33653 } } }
I20260924 03:06:32.604352 17449 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccdab060b5bc4248af1ae024d28208f8 [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:32.604362 17448 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccdab060b5bc4248af1ae024d28208f8 [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:32.604569 17458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:06:32.605113 17458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:06:32.606650 17458 catalog_manager.cc:1347] Generated new cluster ID: 82efb5e03db046f286d169621a0bb325
I20260924 03:06:32.606706 17458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:06:32 dist-test-slave-wgsb krb5kdc[17363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:06:32.616771 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:32.604063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46246 (local address 127.0.98.62:33653)
0924 03:06:32.604613 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:32.604623 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:32.604897 (+ 274us) server_negotiation.cc:407] Connection header received
0924 03:06:32.605023 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:32.605031 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:32.605101 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:32.605233 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:32.606215 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:32.607661 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:32.608791 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:32.609432 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:32.609531 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:32.612441 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:32.612445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:32.612446 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:06:32.612460 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:32.614907 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:32.615363 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:32.615366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:32.615367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:32.615511 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:32.615747 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:32.615748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:32.615749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:32.616125 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:32.616302 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:32.616378 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 03:06:32.616537 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":96,"threads_started":1}
I20260924 03:06:32.625447 17458 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:06:32.626159 17458 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:06:32.630573 17458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccdab060b5bc4248af1ae024d28208f8: Generated new TSK 0
I20260924 03:06:32.630779 17458 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:06:32.668013 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33653
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:06:32.749583 17470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:32.749804 17470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:32.749830 17470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:32.751004 17470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:32.751055 17470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:32.751082 17470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:06:32.752781 17470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:32.753026 17470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:32.753252 17470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:32.758638 17475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:32.758734 17470 server_base.cc:1034] running on GCE node
W20260924 03:06:32.758620 17478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:32.758625 17476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:32.759035 17470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:32.759306 17470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:32.760461 17470 hybrid_clock.cc:648] HybridClock initialized: now 1790219192760450 us; error 50 us; skew 500 ppm
Sep 24 03:06:32 dist-test-slave-wgsb krb5kdc[17363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:32.762989 17470 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:06:32.763417 17470 webserver.cc:460] Webserver started at http://127.0.98.1:42911/ using document root <none> and password file <none>
I20260924 03:06:32.763695 17470 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:32.763764 17470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:32.763919 17470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:32.765064 17470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-0/data/instance:
uuid: "0ae96c816a314e32b9d8a037a9ce1e4c"
format_stamp: "Formatted at 2026-09-24 03:06:32 on dist-test-slave-wgsb"
server_key: "59f2c20ae5bc0c9bfdaefaca82301a41"
server_key_iv: "fb1fc0c9ab0bb156690940927297f238"
server_key_version: "encryptionkey@0"
I20260924 03:06:32.765429 17470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0ae96c816a314e32b9d8a037a9ce1e4c"
format_stamp: "Formatted at 2026-09-24 03:06:32 on dist-test-slave-wgsb"
server_key: "59f2c20ae5bc0c9bfdaefaca82301a41"
server_key_iv: "fb1fc0c9ab0bb156690940927297f238"
server_key_version: "encryptionkey@0"
I20260924 03:06:32.766793 17470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:06:32.767584 17485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:32.767752 17470 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:06:32.767853 17470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "0ae96c816a314e32b9d8a037a9ce1e4c"
format_stamp: "Formatted at 2026-09-24 03:06:32 on dist-test-slave-wgsb"
server_key: "59f2c20ae5bc0c9bfdaefaca82301a41"
server_key_iv: "fb1fc0c9ab0bb156690940927297f238"
server_key_version: "encryptionkey@0"
I20260924 03:06:32.767939 17470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:32.777822 17470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:32.779893 17470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:32.780059 17470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:32.780313 17470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:06:32.780642 17470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:06:32.780695 17470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:32.780751 17470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:06:32.780779 17470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:32.787252 17470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44977
I20260924 03:06:32.787297 17598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44977 every 8 connection(s)
I20260924 03:06:32.787686 17470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:06:32.792973 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 17470
I20260924 03:06:32.793179 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:06:32.793394 392 external_mini_cluster.cc:1428] Setting key 73d8e820cf9626b1d784d0e0a81a306b
Sep 24 03:06:32 dist-test-slave-wgsb krb5kdc[17363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:06:32.798261 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:32.788518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60837 (local address 127.0.98.62:33653)
0924 03:06:32.788605 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:32.788607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:32.788714 (+ 107us) server_negotiation.cc:407] Connection header received
0924 03:06:32.789739 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:32.789739 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:32.789749 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:32.789797 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:32.790383 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:32.790899 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:32.792013 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:32.792243 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:32.792309 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:32.794950 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:32.794952 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:32.794952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:06:32.794960 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:32.796567 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:32.797110 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:32.797111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:32.797112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:32.797169 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:32.797488 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:32.797489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:32.797489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:32.797597 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:32.797646 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:32.798044 (+ 398us) server_negotiation.cc:299] Negotiation successful
0924 03:06:32.798193 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 03:06:32.798502 17601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:32.788450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33653 (local address 127.0.98.1:60837)
0924 03:06:32.788599 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:32.788636 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:32.789613 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:32.789857 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:32.789860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:32.790118 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:06:32.790304 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:32.790307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:32.791043 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:32.791043 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:32.791925 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:32.791927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:32.792326 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:32.792327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:32.792423 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:32.792952 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:06:32.792960 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:06:32.794859 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:06:32.796656 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:06:32.796663 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:06:32.796666 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:06:32.797011 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:06:32.797255 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:06:32.797256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:06:32.797256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:06:32.797412 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:06:32.797662 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:06:32.797669 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:06:32.797853 (+ 184us) client_negotiation.cc:769] Sending connection context
0924 03:06:32.797988 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 03:06:32.798187 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260924 03:06:32.799050 17599 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33653
I20260924 03:06:32.799155 17599 heartbeater.cc:464] Registering TS with master...
I20260924 03:06:32.799357 17599 heartbeater.cc:511] Master 127.0.98.62:33653 requested a full tablet report, sending...
I20260924 03:06:32.799934 17408 ts_manager.cc:194] Registered new tserver with Master: 0ae96c816a314e32b9d8a037a9ce1e4c (127.0.98.1:44977)
I20260924 03:06:32.800549 17408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60837
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:06:32.824777 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33653
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:06:32.913286 17604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:32.913533 17604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:32.913571 17604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:32.914762 17604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:32.914830 17604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:32.914877 17604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:06:32.916431 17604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:32.916695 17604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:32.916920 17604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:32.923874 17610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:32.923854 17609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:32.924010 17612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:32.924154 17604 server_base.cc:1034] running on GCE node
I20260924 03:06:32.924363 17604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:32.924759 17604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:32.925947 17604 hybrid_clock.cc:648] HybridClock initialized: now 1790219192925927 us; error 54 us; skew 500 ppm
Sep 24 03:06:32 dist-test-slave-wgsb krb5kdc[17363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:32.929306 17604 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:06:32.929757 17604 webserver.cc:460] Webserver started at http://127.0.98.2:43177/ using document root <none> and password file <none>
I20260924 03:06:32.929996 17604 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:32.930066 17604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:32.930212 17604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:32.931327 17604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-1/data/instance:
uuid: "02442787d93f4d178afb70339dc02dda"
format_stamp: "Formatted at 2026-09-24 03:06:32 on dist-test-slave-wgsb"
server_key: "bbf1cd183ee5000bdd58756abda11b9e"
server_key_iv: "c784ee394456595f5d712dd915459985"
server_key_version: "encryptionkey@0"
I20260924 03:06:32.931742 17604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-1/wal/instance:
uuid: "02442787d93f4d178afb70339dc02dda"
format_stamp: "Formatted at 2026-09-24 03:06:32 on dist-test-slave-wgsb"
server_key: "bbf1cd183ee5000bdd58756abda11b9e"
server_key_iv: "c784ee394456595f5d712dd915459985"
server_key_version: "encryptionkey@0"
I20260924 03:06:32.933131 17604 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:06:32.934247 17619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:32.934466 17604 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:06:32.934597 17604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "02442787d93f4d178afb70339dc02dda"
format_stamp: "Formatted at 2026-09-24 03:06:32 on dist-test-slave-wgsb"
server_key: "bbf1cd183ee5000bdd58756abda11b9e"
server_key_iv: "c784ee394456595f5d712dd915459985"
server_key_version: "encryptionkey@0"
I20260924 03:06:32.934692 17604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:32.949486 17604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:32.951995 17604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:32.952258 17604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:32.952684 17604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:06:32.953065 17604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:06:32.953130 17604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:32.953181 17604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:06:32.953220 17604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:32.961292 17604 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35109
I20260924 03:06:32.961339 17732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35109 every 8 connection(s)
I20260924 03:06:32.961694 17604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 03:06:32 dist-test-slave-wgsb krb5kdc[17363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:06:32.970640 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 17604
I20260924 03:06:32.970757 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:06:32.970938 392 external_mini_cluster.cc:1428] Setting key 91dbe73214cf2a21f7725f40978b31b4
I20260924 03:06:32.971809 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:32.962452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52253 (local address 127.0.98.62:33653)
0924 03:06:32.962535 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:32.962538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:32.962757 (+ 219us) server_negotiation.cc:407] Connection header received
0924 03:06:32.963767 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:32.963768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:32.963781 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:32.963836 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:32.964465 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:32.964972 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:32.966026 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:32.966236 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:32.966324 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:32.968684 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:32.968686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:32.968686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:06:32.968694 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:32.969923 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:32.970362 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:32.970363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:32.970363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:32.970437 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:32.970779 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:32.970780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:32.970781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:32.971020 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:32.971186 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:32.971564 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 03:06:32.971716 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 03:06:32.972059 17735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:32.962420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33653 (local address 127.0.98.2:52253)
0924 03:06:32.962600 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:32.962642 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:32.963585 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:32.963906 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:32.963911 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:32.964176 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:06:32.964368 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:32.964372 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:32.965067 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:32.965069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:32.965933 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:32.965935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:32.966333 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:32.966334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:32.966464 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:32.966983 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:06:32.966992 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:06:32.968550 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:06:32.970020 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:06:32.970028 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:06:32.970030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:06:32.970267 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:06:32.970562 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:06:32.970563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:06:32.970563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:06:32.970720 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:06:32.971180 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:06:32.971188 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:06:32.971371 (+ 183us) client_negotiation.cc:769] Sending connection context
0924 03:06:32.971512 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 03:06:32.971733 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260924 03:06:32.972855 17733 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33653
I20260924 03:06:32.972967 17733 heartbeater.cc:464] Registering TS with master...
I20260924 03:06:32.973184 17733 heartbeater.cc:511] Master 127.0.98.62:33653 requested a full tablet report, sending...
I20260924 03:06:32.973604 17408 ts_manager.cc:194] Registered new tserver with Master: 02442787d93f4d178afb70339dc02dda (127.0.98.2:35109)
I20260924 03:06:32.973986 17408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52253
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:06:33.001500 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33653
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:06:33.084836 17738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:33.085088 17738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:33.085126 17738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:33.086301 17738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:33.086380 17738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:33.086428 17738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:06:33.088104 17738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:33.088377 17738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:33.088624 17738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:33.094249 17743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:33.094202 17744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:33.094192 17746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:33.094456 17738 server_base.cc:1034] running on GCE node
I20260924 03:06:33.094750 17738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:33.095016 17738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:33.096199 17738 hybrid_clock.cc:648] HybridClock initialized: now 1790219193096170 us; error 69 us; skew 500 ppm
Sep 24 03:06:33 dist-test-slave-wgsb krb5kdc[17363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:33.098729 17738 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:06:33.099301 17738 webserver.cc:460] Webserver started at http://127.0.98.3:34785/ using document root <none> and password file <none>
I20260924 03:06:33.099548 17738 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:33.099640 17738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:33.099817 17738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:33.100842 17738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-2/data/instance:
uuid: "dfec88fd868e42ef839a3e464d299b90"
format_stamp: "Formatted at 2026-09-24 03:06:33 on dist-test-slave-wgsb"
server_key: "e69af9f3d57f2cdd0972b119ec7b9377"
server_key_iv: "f7106093e8fc1e4113d03addedbcd653"
server_key_version: "encryptionkey@0"
I20260924 03:06:33.101141 17738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dfec88fd868e42ef839a3e464d299b90"
format_stamp: "Formatted at 2026-09-24 03:06:33 on dist-test-slave-wgsb"
server_key: "e69af9f3d57f2cdd0972b119ec7b9377"
server_key_iv: "f7106093e8fc1e4113d03addedbcd653"
server_key_version: "encryptionkey@0"
I20260924 03:06:33.102425 17738 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:06:33.103924 17753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:33.104245 17738 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:06:33.104372 17738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "dfec88fd868e42ef839a3e464d299b90"
format_stamp: "Formatted at 2026-09-24 03:06:33 on dist-test-slave-wgsb"
server_key: "e69af9f3d57f2cdd0972b119ec7b9377"
server_key_iv: "f7106093e8fc1e4113d03addedbcd653"
server_key_version: "encryptionkey@0"
I20260924 03:06:33.104451 17738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:33.139714 17738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:33.141834 17738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:33.141994 17738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:33.142246 17738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:06:33.142612 17738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:06:33.142657 17738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:33.142706 17738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:06:33.142733 17738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:33.148943 17738 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41361
I20260924 03:06:33.148993 17866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41361 every 8 connection(s)
I20260924 03:06:33.149339 17738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 03:06:33 dist-test-slave-wgsb krb5kdc[17363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:06:33.157449 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 17738
I20260924 03:06:33.157562 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:06:33.157763 392 external_mini_cluster.cc:1428] Setting key ccb0d3d9ff5506f723589b33c651b95d
I20260924 03:06:33.160595 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:33.150184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52487 (local address 127.0.98.62:33653)
0924 03:06:33.150262 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:33.150264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:33.150355 (+ 91us) server_negotiation.cc:407] Connection header received
0924 03:06:33.151278 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:33.151278 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:33.151288 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:33.151341 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:33.152023 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.152512 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.153587 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.153794 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.153863 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:33.156816 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:33.156818 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:33.156818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:06:33.156826 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:33.158792 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:33.159253 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:33.159254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:33.159254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:33.159316 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:33.159706 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:33.159708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:33.159708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:33.159844 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:33.159916 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:33.160442 (+ 526us) server_negotiation.cc:299] Negotiation successful
0924 03:06:33.160528 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 03:06:33.160755 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:33.150061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33653 (local address 127.0.98.3:52487)
0924 03:06:33.150223 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:33.150272 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:33.151178 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:33.151394 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:33.151397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:33.151741 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:06:33.151943 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:33.151948 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.152616 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.152617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:33.153435 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:33.153437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.153965 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.153966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:33.154180 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:33.154871 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:06:33.154881 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:06:33.156697 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:06:33.158887 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:06:33.158896 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:06:33.158898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:06:33.159155 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:06:33.159415 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:06:33.159417 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:06:33.159417 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:06:33.159553 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:06:33.159945 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:06:33.159951 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:06:33.160092 (+ 141us) client_negotiation.cc:769] Sending connection context
0924 03:06:33.160242 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 03:06:33.160483 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260924 03:06:33.161310 17867 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33653
I20260924 03:06:33.161453 17867 heartbeater.cc:464] Registering TS with master...
I20260924 03:06:33.161682 17867 heartbeater.cc:511] Master 127.0.98.62:33653 requested a full tablet report, sending...
I20260924 03:06:33.162065 17408 ts_manager.cc:194] Registered new tserver with Master: dfec88fd868e42ef839a3e464d299b90 (127.0.98.3:41361)
I20260924 03:06:33.162442 17408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52487
I20260924 03:06:33.169904 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:06:33.177845 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:33.171478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46254 (local address 127.0.98.62:33653)
0924 03:06:33.171547 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:33.171549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:33.171666 (+ 117us) server_negotiation.cc:407] Connection header received
0924 03:06:33.171749 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:33.171749 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:33.171759 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:33.171831 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:33.172416 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.172946 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.173714 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.173905 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.173963 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:33.175206 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:33.175208 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:33.175208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:06:33.175215 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:33.176886 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:33.177195 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:33.177196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:33.177196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:33.177249 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:33.177397 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:33.177397 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:33.177397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:33.177538 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:33.177612 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:33.177698 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 03:06:33.177771 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 03:06:33.179522 17408 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:06:33.181149 17408 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:06:33.187633 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:33.183411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35109 (local address 127.0.0.1:60904)
0924 03:06:33.183868 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:33.183879 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:33.183938 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:33.184246 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:33.184250 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:33.184252 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:33.184321 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:33.184323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.185675 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.185676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:33.186509 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:33.186511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.187294 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.187295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:33.187367 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:33.187382 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 03:06:33.187454 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 03:06:33.187504 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":33,"thread_start_us":96,"threads_started":1}
I20260924 03:06:33.187650 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:33.183886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60904 (local address 127.0.98.2:35109)
0924 03:06:33.184048 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:33.184052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:33.184060 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:06:33.184090 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:33.184092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:33.184103 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:33.184193 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:33.184412 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.185560 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.186612 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.187215 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.187288 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:33.187344 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:33.187484 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 03:06:33.187527 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260924 03:06:33.188153 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:33.183397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41361 (local address 127.0.0.1:43290)
0924 03:06:33.183611 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:33.183652 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:33.183720 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:33.184342 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:33.184343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:33.184345 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:33.184388 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:33.184389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.185647 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.185647 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:33.186863 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:33.186865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.187905 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.187905 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:33.187967 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:33.187971 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:06:33.188031 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 03:06:33.188077 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":54,"thread_start_us":112,"threads_started":1}
I20260924 03:06:33.188172 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:33.183671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43290 (local address 127.0.98.3:41361)
0924 03:06:33.184176 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:33.184182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:33.184194 (+ 12us) server_negotiation.cc:407] Connection header received
0924 03:06:33.184237 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:33.184240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:33.184256 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:33.184338 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:33.184476 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.185536 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.186949 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.187680 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.187892 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:33.187958 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:33.188052 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 03:06:33.188091 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":103,"threads_started":1}
I20260924 03:06:33.188333 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:33.183399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44977 (local address 127.0.0.1:39482)
0924 03:06:33.183616 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:33.183652 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:33.183720 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:33.184636 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:33.184637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:33.184639 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:33.184685 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:33.184687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.185526 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.185527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:33.187084 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:33.187086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.188131 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.188132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:33.188191 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:33.188196 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:06:33.188229 (+ 33us) client_negotiation.cc:240] Negotiation successful
0924 03:06:33.188271 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":68,"thread_start_us":44,"threads_started":1}
I20260924 03:06:33.188568 17667 tablet_service.cc:1467] Processing CreateTablet for tablet fa769cb1a21b4e67853150c756daa587 (DEFAULT_TABLE table=test-table [id=c1ee34c3a46547e48717c371ab01ad79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:06:33.188717 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:33.183696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39482 (local address 127.0.98.1:44977)
0924 03:06:33.184479 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:33.184483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:33.184492 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:06:33.184525 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:33.184527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:33.184549 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:33.184636 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:33.184742 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.185422 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.187163 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.188034 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.188499 (+ 465us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:33.188561 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:33.188586 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 03:06:33.188626 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":83,"threads_started":1}
I20260924 03:06:33.188967 17667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa769cb1a21b4e67853150c756daa587. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:33.188946 17801 tablet_service.cc:1467] Processing CreateTablet for tablet fa769cb1a21b4e67853150c756daa587 (DEFAULT_TABLE table=test-table [id=c1ee34c3a46547e48717c371ab01ad79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:06:33.189230 17801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa769cb1a21b4e67853150c756daa587. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:33.189227 17533 tablet_service.cc:1467] Processing CreateTablet for tablet fa769cb1a21b4e67853150c756daa587 (DEFAULT_TABLE table=test-table [id=c1ee34c3a46547e48717c371ab01ad79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:06:33.189450 17533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa769cb1a21b4e67853150c756daa587. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:33.192121 17884 tablet_bootstrap.cc:492] T fa769cb1a21b4e67853150c756daa587 P 0ae96c816a314e32b9d8a037a9ce1e4c: Bootstrap starting.
I20260924 03:06:33.192550 17885 tablet_bootstrap.cc:492] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda: Bootstrap starting.
I20260924 03:06:33.192798 17884 tablet_bootstrap.cc:654] T fa769cb1a21b4e67853150c756daa587 P 0ae96c816a314e32b9d8a037a9ce1e4c: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:33.193219 17885 tablet_bootstrap.cc:654] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:33.193400 17884 log.cc:824] T fa769cb1a21b4e67853150c756daa587 P 0ae96c816a314e32b9d8a037a9ce1e4c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:33.193764 17885 log.cc:824] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:33.194077 17886 tablet_bootstrap.cc:492] T fa769cb1a21b4e67853150c756daa587 P dfec88fd868e42ef839a3e464d299b90: Bootstrap starting.
I20260924 03:06:33.194273 17884 tablet_bootstrap.cc:492] T fa769cb1a21b4e67853150c756daa587 P 0ae96c816a314e32b9d8a037a9ce1e4c: No bootstrap required, opened a new log
I20260924 03:06:33.194355 17884 ts_tablet_manager.cc:1397] T fa769cb1a21b4e67853150c756daa587 P 0ae96c816a314e32b9d8a037a9ce1e4c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:06:33.194481 17885 tablet_bootstrap.cc:492] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda: No bootstrap required, opened a new log
I20260924 03:06:33.194571 17885 ts_tablet_manager.cc:1397] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:06:33.194720 17886 tablet_bootstrap.cc:654] T fa769cb1a21b4e67853150c756daa587 P dfec88fd868e42ef839a3e464d299b90: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:33.195287 17886 log.cc:824] T fa769cb1a21b4e67853150c756daa587 P dfec88fd868e42ef839a3e464d299b90: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:33.196072 17886 tablet_bootstrap.cc:492] T fa769cb1a21b4e67853150c756daa587 P dfec88fd868e42ef839a3e464d299b90: No bootstrap required, opened a new log
I20260924 03:06:33.196143 17886 ts_tablet_manager.cc:1397] T fa769cb1a21b4e67853150c756daa587 P dfec88fd868e42ef839a3e464d299b90: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:06:33.196529 17885 raft_consensus.cc:348] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae96c816a314e32b9d8a037a9ce1e4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44977 } } peers { permanent_uuid: "dfec88fd868e42ef839a3e464d299b90" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41361 } } peers { permanent_uuid: "02442787d93f4d178afb70339dc02dda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35109 } }
I20260924 03:06:33.196720 17885 raft_consensus.cc:374] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:33.196767 17885 raft_consensus.cc:729] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02442787d93f4d178afb70339dc02dda, State: Initialized, Role: FOLLOWER
I20260924 03:06:33.196713 17884 raft_consensus.cc:348] T fa769cb1a21b4e67853150c756daa587 P 0ae96c816a314e32b9d8a037a9ce1e4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae96c816a314e32b9d8a037a9ce1e4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44977 } } peers { permanent_uuid: "dfec88fd868e42ef839a3e464d299b90" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41361 } } peers { permanent_uuid: "02442787d93f4d178afb70339dc02dda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35109 } }
I20260924 03:06:33.196885 17884 raft_consensus.cc:374] T fa769cb1a21b4e67853150c756daa587 P 0ae96c816a314e32b9d8a037a9ce1e4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:33.196882 17885 consensus_queue.cc:260] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae96c816a314e32b9d8a037a9ce1e4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44977 } } peers { permanent_uuid: "dfec88fd868e42ef839a3e464d299b90" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41361 } } peers { permanent_uuid: "02442787d93f4d178afb70339dc02dda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35109 } }
I20260924 03:06:33.196933 17884 raft_consensus.cc:729] T fa769cb1a21b4e67853150c756daa587 P 0ae96c816a314e32b9d8a037a9ce1e4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ae96c816a314e32b9d8a037a9ce1e4c, State: Initialized, Role: FOLLOWER
I20260924 03:06:33.197060 17884 consensus_queue.cc:260] T fa769cb1a21b4e67853150c756daa587 P 0ae96c816a314e32b9d8a037a9ce1e4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae96c816a314e32b9d8a037a9ce1e4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44977 } } peers { permanent_uuid: "dfec88fd868e42ef839a3e464d299b90" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41361 } } peers { permanent_uuid: "02442787d93f4d178afb70339dc02dda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35109 } }
I20260924 03:06:33.197157 17733 heartbeater.cc:503] Master 127.0.98.62:33653 was elected leader, sending a full tablet report...
I20260924 03:06:33.197167 17885 ts_tablet_manager.cc:1428] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:06:33.197332 17884 ts_tablet_manager.cc:1428] T fa769cb1a21b4e67853150c756daa587 P 0ae96c816a314e32b9d8a037a9ce1e4c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:06:33.197438 17599 heartbeater.cc:503] Master 127.0.98.62:33653 was elected leader, sending a full tablet report...
I20260924 03:06:33.198246 17886 raft_consensus.cc:348] T fa769cb1a21b4e67853150c756daa587 P dfec88fd868e42ef839a3e464d299b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae96c816a314e32b9d8a037a9ce1e4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44977 } } peers { permanent_uuid: "dfec88fd868e42ef839a3e464d299b90" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41361 } } peers { permanent_uuid: "02442787d93f4d178afb70339dc02dda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35109 } }
I20260924 03:06:33.198365 17886 raft_consensus.cc:374] T fa769cb1a21b4e67853150c756daa587 P dfec88fd868e42ef839a3e464d299b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:33.198411 17886 raft_consensus.cc:729] T fa769cb1a21b4e67853150c756daa587 P dfec88fd868e42ef839a3e464d299b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfec88fd868e42ef839a3e464d299b90, State: Initialized, Role: FOLLOWER
I20260924 03:06:33.198532 17886 consensus_queue.cc:260] T fa769cb1a21b4e67853150c756daa587 P dfec88fd868e42ef839a3e464d299b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae96c816a314e32b9d8a037a9ce1e4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44977 } } peers { permanent_uuid: "dfec88fd868e42ef839a3e464d299b90" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41361 } } peers { permanent_uuid: "02442787d93f4d178afb70339dc02dda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35109 } }
I20260924 03:06:33.198814 17886 ts_tablet_manager.cc:1428] T fa769cb1a21b4e67853150c756daa587 P dfec88fd868e42ef839a3e464d299b90: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:06:33.198838 17867 heartbeater.cc:503] Master 127.0.98.62:33653 was elected leader, sending a full tablet report...
W20260924 03:06:33.212268 17734 tablet.cc:2369] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:06:33.288501 17600 tablet.cc:2369] T fa769cb1a21b4e67853150c756daa587 P 0ae96c816a314e32b9d8a037a9ce1e4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:06:33.399936 17868 tablet.cc:2369] T fa769cb1a21b4e67853150c756daa587 P dfec88fd868e42ef839a3e464d299b90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:06:33.463054 17890 raft_consensus.cc:482] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:06:33.463239 17890 raft_consensus.cc:504] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae96c816a314e32b9d8a037a9ce1e4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44977 } } peers { permanent_uuid: "dfec88fd868e42ef839a3e464d299b90" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41361 } } peers { permanent_uuid: "02442787d93f4d178afb70339dc02dda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35109 } }
I20260924 03:06:33.463793 17890 leader_election.cc:288] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ae96c816a314e32b9d8a037a9ce1e4c (127.0.98.1:44977), dfec88fd868e42ef839a3e464d299b90 (127.0.98.3:41361)
I20260924 03:06:33.468055 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:33.464037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41361 (local address 127.0.98.2:49729)
0924 03:06:33.464238 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:33.464247 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:33.464316 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:33.464464 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:33.464465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:33.464466 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:33.464525 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:33.464526 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.465444 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.465445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:33.466847 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:33.466850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.467739 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.467740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:33.467832 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:33.467839 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 03:06:33.467910 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 03:06:33.467952 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":106,"thread_start_us":78,"threads_started":1}
I20260924 03:06:33.468072 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:33.464058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46921 (local address 127.0.98.1:44977)
0924 03:06:33.464106 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:33.464108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:33.464149 (+ 41us) server_negotiation.cc:407] Connection header received
0924 03:06:33.464254 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:33.464255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:33.464266 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:33.464350 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:33.464559 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.465338 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.466931 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.467658 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.467726 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:33.467735 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:33.467953 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 03:06:33.468003 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 03:06:33.468072 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:33.464131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49729 (local address 127.0.98.3:41361)
0924 03:06:33.464186 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:33.464188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:33.464326 (+ 138us) server_negotiation.cc:407] Connection header received
0924 03:06:33.464387 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:33.464388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:33.464398 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:33.464450 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:33.464629 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.465340 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.466935 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.467632 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.467721 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:33.467735 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:33.467954 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 03:06:33.468001 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 03:06:33.468108 17735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:33.464027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44977 (local address 127.0.98.2:46921)
0924 03:06:33.464091 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:33.464103 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:33.464164 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:33.464371 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:33.464372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:33.464374 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:33.464474 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:33.464476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.465444 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.465445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:33.466847 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:33.466850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.467739 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.467740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:33.467832 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:33.467838 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 03:06:33.467906 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 03:06:33.467952 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":57}
I20260924 03:06:33.468639 17553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa769cb1a21b4e67853150c756daa587" candidate_uuid: "02442787d93f4d178afb70339dc02dda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ae96c816a314e32b9d8a037a9ce1e4c" is_pre_election: true
I20260924 03:06:33.468775 17553 raft_consensus.cc:2454] T fa769cb1a21b4e67853150c756daa587 P 0ae96c816a314e32b9d8a037a9ce1e4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02442787d93f4d178afb70339dc02dda in term 0.
I20260924 03:06:33.469010 17622 leader_election.cc:302] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02442787d93f4d178afb70339dc02dda, 0ae96c816a314e32b9d8a037a9ce1e4c; no voters:
I20260924 03:06:33.469050 17821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa769cb1a21b4e67853150c756daa587" candidate_uuid: "02442787d93f4d178afb70339dc02dda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfec88fd868e42ef839a3e464d299b90" is_pre_election: true
I20260924 03:06:33.469183 17821 raft_consensus.cc:2454] T fa769cb1a21b4e67853150c756daa587 P dfec88fd868e42ef839a3e464d299b90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02442787d93f4d178afb70339dc02dda in term 0.
I20260924 03:06:33.469252 17890 raft_consensus.cc:2781] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:06:33.469345 17890 raft_consensus.cc:482] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:06:33.469409 17890 raft_consensus.cc:3037] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:33.470274 17890 raft_consensus.cc:504] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae96c816a314e32b9d8a037a9ce1e4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44977 } } peers { permanent_uuid: "dfec88fd868e42ef839a3e464d299b90" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41361 } } peers { permanent_uuid: "02442787d93f4d178afb70339dc02dda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35109 } }
I20260924 03:06:33.470482 17890 leader_election.cc:288] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda [CANDIDATE]: Term 1 election: Requested vote from peers 0ae96c816a314e32b9d8a037a9ce1e4c (127.0.98.1:44977), dfec88fd868e42ef839a3e464d299b90 (127.0.98.3:41361)
I20260924 03:06:33.470567 17553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa769cb1a21b4e67853150c756daa587" candidate_uuid: "02442787d93f4d178afb70339dc02dda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ae96c816a314e32b9d8a037a9ce1e4c"
I20260924 03:06:33.470636 17553 raft_consensus.cc:3037] T fa769cb1a21b4e67853150c756daa587 P 0ae96c816a314e32b9d8a037a9ce1e4c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:33.470646 17821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa769cb1a21b4e67853150c756daa587" candidate_uuid: "02442787d93f4d178afb70339dc02dda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfec88fd868e42ef839a3e464d299b90"
I20260924 03:06:33.470721 17821 raft_consensus.cc:3037] T fa769cb1a21b4e67853150c756daa587 P dfec88fd868e42ef839a3e464d299b90 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:33.471311 17553 raft_consensus.cc:2454] T fa769cb1a21b4e67853150c756daa587 P 0ae96c816a314e32b9d8a037a9ce1e4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02442787d93f4d178afb70339dc02dda in term 1.
I20260924 03:06:33.471513 17622 leader_election.cc:302] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02442787d93f4d178afb70339dc02dda, 0ae96c816a314e32b9d8a037a9ce1e4c; no voters:
I20260924 03:06:33.471639 17890 raft_consensus.cc:2781] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:06:33.471688 17821 raft_consensus.cc:2454] T fa769cb1a21b4e67853150c756daa587 P dfec88fd868e42ef839a3e464d299b90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02442787d93f4d178afb70339dc02dda in term 1.
I20260924 03:06:33.471808 17890 raft_consensus.cc:686] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda [term 1 LEADER]: Becoming Leader. State: Replica: 02442787d93f4d178afb70339dc02dda, State: Running, Role: LEADER
I20260924 03:06:33.471923 17890 consensus_queue.cc:237] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae96c816a314e32b9d8a037a9ce1e4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44977 } } peers { permanent_uuid: "dfec88fd868e42ef839a3e464d299b90" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41361 } } peers { permanent_uuid: "02442787d93f4d178afb70339dc02dda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35109 } }
I20260924 03:06:33.472735 17407 catalog_manager.cc:5495] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda reported cstate change: term changed from 0 to 1, leader changed from <none> to 02442787d93f4d178afb70339dc02dda (127.0.98.2). New cstate: current_term: 1 leader_uuid: "02442787d93f4d178afb70339dc02dda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae96c816a314e32b9d8a037a9ce1e4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfec88fd868e42ef839a3e464d299b90" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02442787d93f4d178afb70339dc02dda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35109 } health_report { overall_health: HEALTHY } } }
I20260924 03:06:33.541982 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:33.539340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60910 (local address 127.0.98.2:35109)
0924 03:06:33.539408 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:33.539410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:33.539418 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:06:33.539500 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:33.539501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:33.539520 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:33.539610 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:06:33.539807 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.540314 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.541153 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.541343 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.541406 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:33.541583 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:06:33.541699 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:06:33.541777 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:33.541813 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 03:06:33.541833 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 03:06:33.544261 17821 raft_consensus.cc:1261] T fa769cb1a21b4e67853150c756daa587 P dfec88fd868e42ef839a3e464d299b90 [term 1 FOLLOWER]: Refusing update from remote peer 02442787d93f4d178afb70339dc02dda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:06:33.544449 17553 raft_consensus.cc:1261] T fa769cb1a21b4e67853150c756daa587 P 0ae96c816a314e32b9d8a037a9ce1e4c [term 1 FOLLOWER]: Refusing update from remote peer 02442787d93f4d178afb70339dc02dda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:06:33.544790 17894 consensus_queue.cc:1035] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfec88fd868e42ef839a3e464d299b90" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:06:33.544945 17890 consensus_queue.cc:1035] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ae96c816a314e32b9d8a037a9ce1e4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:06:33.547343 17897 mvcc.cc:205] Tried to move back new op lower bound from 7332737816753504256 to 7332737816461672448. Current Snapshot: MvccSnapshot[applied={T|T < 7332737816753504256}]
I20260924 03:06:33.547441 17899 mvcc.cc:205] Tried to move back new op lower bound from 7332737816753504256 to 7332737816461672448. Current Snapshot: MvccSnapshot[applied={T|T < 7332737816753504256}]
I20260924 03:06:33.547515 17898 mvcc.cc:205] Tried to move back new op lower bound from 7332737816753504256 to 7332737816461672448. Current Snapshot: MvccSnapshot[applied={T|T < 7332737816753504256}]
I20260924 03:06:33.550056 17408 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:06:33.550177 17408 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:06:33.550956 17408 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ccdab060b5bc4248af1ae024d28208f8: Sending DeleteTablet for 3 replicas of tablet fa769cb1a21b4e67853150c756daa587
I20260924 03:06:33.551329 17801 tablet_service.cc:1514] Processing DeleteTablet for tablet fa769cb1a21b4e67853150c756daa587 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:06:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43290
I20260924 03:06:33.551633 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 17470
I20260924 03:06:33.552592 17667 tablet_service.cc:1514] Processing DeleteTablet for tablet fa769cb1a21b4e67853150c756daa587 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:06:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60904
I20260924 03:06:33.552981 17908 tablet_replica.cc:330] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda: stopping tablet replica
I20260924 03:06:33.553149 17908 raft_consensus.cc:2229] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:06:33.553305 17908 raft_consensus.cc:2258] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:06:33.555676 17907 tablet_replica.cc:330] T fa769cb1a21b4e67853150c756daa587 P dfec88fd868e42ef839a3e464d299b90: stopping tablet replica
I20260924 03:06:33.555845 17907 raft_consensus.cc:2229] T fa769cb1a21b4e67853150c756daa587 P dfec88fd868e42ef839a3e464d299b90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:06:33.555984 17907 raft_consensus.cc:2258] T fa769cb1a21b4e67853150c756daa587 P dfec88fd868e42ef839a3e464d299b90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:06:33.556272 17908 ts_tablet_manager.cc:1905] T fa769cb1a21b4e67853150c756daa587 P 02442787d93f4d178afb70339dc02dda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:06:33.556531 17907 ts_tablet_manager.cc:1905] T fa769cb1a21b4e67853150c756daa587 P dfec88fd868e42ef839a3e464d299b90: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 03:06:33.557432 17394 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44977 (error 108)
I20260924 03:06:33.558229 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 17604
I20260924 03:06:33.559082 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:33.558948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44977 (local address 127.0.0.1:39498)
0924 03:06:33.558997 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:33.559033 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 03:06:33.559191 17394 catalog_manager.cc:4584] TS 0ae96c816a314e32b9d8a037a9ce1e4c (127.0.98.1:44977): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fa769cb1a21b4e67853150c756daa587: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44977: connect: Connection refused (error 111)
I20260924 03:06:33.563971 17907 ts_tablet_manager.cc:1918] T fa769cb1a21b4e67853150c756daa587 P dfec88fd868e42ef839a3e464d299b90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:06:33.564133 17907 log.cc:1196] T fa769cb1a21b4e67853150c756daa587 P dfec88fd868e42ef839a3e464d299b90: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218948286999-392-0/minicluster-data/ts-2/wal/wals/fa769cb1a21b4e67853150c756daa587
I20260924 03:06:33.564164 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:33.564028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35109 (local address 127.0.0.1:60920)
0924 03:06:33.564088 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:33.564117 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 03:06:33.564406 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 17738
I20260924 03:06:33.568965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 17376
I20260924 03:06:33.569032 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:33.568894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41361 (local address 127.0.0.1:43292)
0924 03:06:33.568961 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:33.568993 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:41361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
2026-09-24T03:06:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2341 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T03:06:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:06:33Z Disabled control of system clock
I20260924 03:06:33.597600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33675
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:06:33.679045 17913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:33.679278 17913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:33.679301 17913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:33.680469 17913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:06:33.680511 17913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:33.680524 17913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:33.680537 17913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:06:33.680549 17913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:06:33.681962 17913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33675
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:33.682174 17913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:33.682413 17913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:33.687515 17919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:33.687578 17918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:33.687646 17921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:33.687834 17913 server_base.cc:1034] running on GCE node
I20260924 03:06:33.688038 17913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:33.688282 17913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:33.689465 17913 hybrid_clock.cc:648] HybridClock initialized: now 1790219193689440 us; error 36 us; skew 500 ppm
I20260924 03:06:33.690886 17913 webserver.cc:460] Webserver started at http://127.0.98.62:44797/ using document root <none> and password file <none>
I20260924 03:06:33.691162 17913 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:33.691236 17913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:33.691392 17913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:33.692628 17913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "281f4d58a73d4931be3a572428ccd254"
format_stamp: "Formatted at 2026-09-24 03:06:33 on dist-test-slave-wgsb"
server_key: "9eedec1c2c62764931314df70f618894"
server_key_iv: "f96be1db8db1e46c897b8005a984a8fb"
server_key_version: "encryptionkey@0"
I20260924 03:06:33.693003 17913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "281f4d58a73d4931be3a572428ccd254"
format_stamp: "Formatted at 2026-09-24 03:06:33 on dist-test-slave-wgsb"
server_key: "9eedec1c2c62764931314df70f618894"
server_key_iv: "f96be1db8db1e46c897b8005a984a8fb"
server_key_version: "encryptionkey@0"
I20260924 03:06:33.694432 17913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:06:33.695302 17927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:33.695585 17913 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:06:33.695765 17913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "281f4d58a73d4931be3a572428ccd254"
format_stamp: "Formatted at 2026-09-24 03:06:33 on dist-test-slave-wgsb"
server_key: "9eedec1c2c62764931314df70f618894"
server_key_iv: "f96be1db8db1e46c897b8005a984a8fb"
server_key_version: "encryptionkey@0"
I20260924 03:06:33.695871 17913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:33.717712 17913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:33.718101 17913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:33.718281 17913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:33.722892 17913 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33675
I20260924 03:06:33.722980 17979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33675 every 8 connection(s)
I20260924 03:06:33.723376 17913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:06:33.724484 17980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:33.726936 17980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 281f4d58a73d4931be3a572428ccd254: Bootstrap starting.
I20260924 03:06:33.727520 17980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 281f4d58a73d4931be3a572428ccd254: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:33.727941 17980 log.cc:824] T 00000000000000000000000000000000 P 281f4d58a73d4931be3a572428ccd254: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:33.728713 17980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 281f4d58a73d4931be3a572428ccd254: No bootstrap required, opened a new log
I20260924 03:06:33.730283 17980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 281f4d58a73d4931be3a572428ccd254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "281f4d58a73d4931be3a572428ccd254" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33675 } }
I20260924 03:06:33.730422 17980 raft_consensus.cc:374] T 00000000000000000000000000000000 P 281f4d58a73d4931be3a572428ccd254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:33.730444 17980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 281f4d58a73d4931be3a572428ccd254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 281f4d58a73d4931be3a572428ccd254, State: Initialized, Role: FOLLOWER
I20260924 03:06:33.730582 17980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 281f4d58a73d4931be3a572428ccd254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "281f4d58a73d4931be3a572428ccd254" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33675 } }
I20260924 03:06:33.730660 17980 raft_consensus.cc:388] T 00000000000000000000000000000000 P 281f4d58a73d4931be3a572428ccd254 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:06:33.730692 17980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 281f4d58a73d4931be3a572428ccd254 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:06:33.730727 17980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 281f4d58a73d4931be3a572428ccd254 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:33.731218 17980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 281f4d58a73d4931be3a572428ccd254 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "281f4d58a73d4931be3a572428ccd254" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33675 } }
I20260924 03:06:33.731303 17980 leader_election.cc:302] T 00000000000000000000000000000000 P 281f4d58a73d4931be3a572428ccd254 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 281f4d58a73d4931be3a572428ccd254; no voters:
I20260924 03:06:33.731437 17980 leader_election.cc:288] T 00000000000000000000000000000000 P 281f4d58a73d4931be3a572428ccd254 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:06:33.731542 17983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 281f4d58a73d4931be3a572428ccd254 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:06:33.731748 17983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 281f4d58a73d4931be3a572428ccd254 [term 1 LEADER]: Becoming Leader. State: Replica: 281f4d58a73d4931be3a572428ccd254, State: Running, Role: LEADER
I20260924 03:06:33.731818 17980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 281f4d58a73d4931be3a572428ccd254 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:06:33.731849 17983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 281f4d58a73d4931be3a572428ccd254 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "281f4d58a73d4931be3a572428ccd254" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33675 } }
I20260924 03:06:33.732293 17985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 281f4d58a73d4931be3a572428ccd254 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 281f4d58a73d4931be3a572428ccd254. Latest consensus state: current_term: 1 leader_uuid: "281f4d58a73d4931be3a572428ccd254" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "281f4d58a73d4931be3a572428ccd254" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33675 } } }
I20260924 03:06:33.732378 17985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 281f4d58a73d4931be3a572428ccd254 [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:33.732466 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 17913
I20260924 03:06:33.732575 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:06:33.732787 392 external_mini_cluster.cc:1428] Setting key b4c7c63606485c631b1b67dd254ba2be
I20260924 03:06:33.732793 17990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:06:33.733306 17990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:06:33.733579 17984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 281f4d58a73d4931be3a572428ccd254 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "281f4d58a73d4931be3a572428ccd254" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "281f4d58a73d4931be3a572428ccd254" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33675 } } }
I20260924 03:06:33.733734 17984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 281f4d58a73d4931be3a572428ccd254 [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:33.735087 17990 catalog_manager.cc:1347] Generated new cluster ID: 2b21a788aa8e4db3aec953dc991d9155
I20260924 03:06:33.735143 17990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:06:33.738611 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:33.733482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34282 (local address 127.0.98.62:33675)
0924 03:06:33.733992 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:33.733999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:33.734022 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:06:33.734112 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:33.734117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:33.734223 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:33.734359 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:33.734597 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.735770 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.736552 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.737050 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.737190 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:33.737940 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:33.737944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:33.737944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:33.738132 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:33.738163 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:33.738177 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:33.738251 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:33.738326 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 03:06:33.738404 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":76,"threads_started":1}
I20260924 03:06:33.759054 17990 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:06:33.759765 17990 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:06:33.771052 17990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 281f4d58a73d4931be3a572428ccd254: Generated new TSK 0
I20260924 03:06:33.771320 17990 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:06:33.778466 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33675
--builtin_ntp_servers=127.0.98.20:37181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:06:33.865424 18004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:33.865681 18004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:33.865736 18004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:33.866904 18004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:33.866966 18004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:33.867012 18004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:06:33.868458 18004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:33.868726 18004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:33.869029 18004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:06:33.875118 18004 server_base.cc:1034] running on GCE node
W20260924 03:06:33.875025 18009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:33.875029 18010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:33.875061 18012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:33.875489 18004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:33.875767 18004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:33.876938 18004 hybrid_clock.cc:648] HybridClock initialized: now 1790219193876923 us; error 48 us; skew 500 ppm
I20260924 03:06:33.878149 18004 webserver.cc:460] Webserver started at http://127.0.98.1:42847/ using document root <none> and password file <none>
I20260924 03:06:33.878438 18004 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:33.878509 18004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:33.878664 18004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:33.879832 18004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-0/data/instance:
uuid: "0a27c7844ab549e588dbaec4b0fa780f"
format_stamp: "Formatted at 2026-09-24 03:06:33 on dist-test-slave-wgsb"
server_key: "ea9b4ca1ed804d512950e31b6a973ef5"
server_key_iv: "f24d6ae2341d87e4d36051a2b2b95786"
server_key_version: "encryptionkey@0"
I20260924 03:06:33.880195 18004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0a27c7844ab549e588dbaec4b0fa780f"
format_stamp: "Formatted at 2026-09-24 03:06:33 on dist-test-slave-wgsb"
server_key: "ea9b4ca1ed804d512950e31b6a973ef5"
server_key_iv: "f24d6ae2341d87e4d36051a2b2b95786"
server_key_version: "encryptionkey@0"
I20260924 03:06:33.881531 18004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:06:33.882465 18018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:33.882699 18004 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:06:33.882822 18004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "0a27c7844ab549e588dbaec4b0fa780f"
format_stamp: "Formatted at 2026-09-24 03:06:33 on dist-test-slave-wgsb"
server_key: "ea9b4ca1ed804d512950e31b6a973ef5"
server_key_iv: "f24d6ae2341d87e4d36051a2b2b95786"
server_key_version: "encryptionkey@0"
I20260924 03:06:33.882915 18004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:33.900043 18004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:33.900382 18004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:33.900501 18004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:33.900705 18004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:06:33.901242 18004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:06:33.901331 18004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:33.901381 18004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:06:33.901412 18004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:33.907588 18004 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42473
I20260924 03:06:33.907658 18131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42473 every 8 connection(s)
I20260924 03:06:33.907990 18004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:06:33.913470 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:33.908803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56017 (local address 127.0.98.62:33675)
0924 03:06:33.908876 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:33.908878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:33.909342 (+ 464us) server_negotiation.cc:407] Connection header received
0924 03:06:33.909668 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:33.909669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:33.909683 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:33.909753 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:33.910418 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.910919 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.912101 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.912293 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.912360 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:33.913233 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:33.913235 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:33.913235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:33.913243 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:33.913258 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:33.913262 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:33.913311 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:33.913383 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 03:06:33.913408 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 03:06:33.913705 18134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:33.908803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33675 (local address 127.0.98.1:56017)
0924 03:06:33.908997 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:33.909039 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:33.909569 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:33.909805 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:33.909809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:33.909817 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:06:33.910329 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:33.910333 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.911005 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.911006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:33.911956 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:33.911959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:33.912377 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:33.912378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:33.912488 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:33.913084 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:06:33.913084 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:06:33.913114 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:06:33.913137 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:06:33.913309 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:06:33.913311 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:06:33.913314 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:06:33.913361 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 03:06:33.913442 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
I20260924 03:06:33.914266 18132 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33675
I20260924 03:06:33.914359 18132 heartbeater.cc:464] Registering TS with master...
I20260924 03:06:33.914559 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 18004
I20260924 03:06:33.914572 18132 heartbeater.cc:511] Master 127.0.98.62:33675 requested a full tablet report, sending...
I20260924 03:06:33.914665 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:06:33.914903 392 external_mini_cluster.cc:1428] Setting key c0b1668bc7aa677b037ac93140bd14df
I20260924 03:06:33.915191 17944 ts_manager.cc:194] Registered new tserver with Master: 0a27c7844ab549e588dbaec4b0fa780f (127.0.98.1:42473)
I20260924 03:06:33.915980 17944 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:56017
I20260924 03:06:33.916002 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33675
--builtin_ntp_servers=127.0.98.20:37181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:06:33.997501 18135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:33.997767 18135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:33.997805 18135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:33.999110 18135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:33.999171 18135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:33.999218 18135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:06:34.000785 18135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:34.001066 18135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:34.001381 18135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:34.006666 18141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:34.006680 18143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:34.006675 18140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:34.006740 18135 server_base.cc:1034] running on GCE node
I20260924 03:06:34.007009 18135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:34.007275 18135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:34.008430 18135 hybrid_clock.cc:648] HybridClock initialized: now 1790219194008415 us; error 36 us; skew 500 ppm
I20260924 03:06:34.009621 18135 webserver.cc:460] Webserver started at http://127.0.98.2:39197/ using document root <none> and password file <none>
I20260924 03:06:34.009860 18135 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:34.009925 18135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:34.010084 18135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:34.011143 18135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-1/data/instance:
uuid: "73e0fa2aa8654234a84a74878a3cd4db"
format_stamp: "Formatted at 2026-09-24 03:06:34 on dist-test-slave-wgsb"
server_key: "77deea00c37c6d870ea550b48f4e82b1"
server_key_iv: "ee4cb7e54c4069b209b7bbe179d6f241"
server_key_version: "encryptionkey@0"
I20260924 03:06:34.011487 18135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-1/wal/instance:
uuid: "73e0fa2aa8654234a84a74878a3cd4db"
format_stamp: "Formatted at 2026-09-24 03:06:34 on dist-test-slave-wgsb"
server_key: "77deea00c37c6d870ea550b48f4e82b1"
server_key_iv: "ee4cb7e54c4069b209b7bbe179d6f241"
server_key_version: "encryptionkey@0"
I20260924 03:06:34.012786 18135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:06:34.013566 18149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:34.013814 18135 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:06:34.013936 18135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "73e0fa2aa8654234a84a74878a3cd4db"
format_stamp: "Formatted at 2026-09-24 03:06:34 on dist-test-slave-wgsb"
server_key: "77deea00c37c6d870ea550b48f4e82b1"
server_key_iv: "ee4cb7e54c4069b209b7bbe179d6f241"
server_key_version: "encryptionkey@0"
I20260924 03:06:34.014017 18135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:34.030777 18135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:34.031116 18135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:34.031271 18135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:34.031582 18135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:06:34.032048 18135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:06:34.032114 18135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:34.032161 18135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:06:34.032194 18135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:34.038784 18135 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45339
I20260924 03:06:34.038834 18262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45339 every 8 connection(s)
I20260924 03:06:34.039197 18135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:06:34.040820 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 18135
I20260924 03:06:34.040933 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:06:34.041170 392 external_mini_cluster.cc:1428] Setting key 5df4c02ae95647ad248f7a9ea564a89b
I20260924 03:06:34.042294 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33675
--builtin_ntp_servers=127.0.98.20:37181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:06:34.045975 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:34.039976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44863 (local address 127.0.98.62:33675)
0924 03:06:34.040054 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:34.040056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:34.040309 (+ 253us) server_negotiation.cc:407] Connection header received
0924 03:06:34.040860 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:34.040861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:34.040877 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:34.040968 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:34.042221 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.042826 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.044300 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.044520 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.044615 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:34.045626 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:34.045628 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:34.045628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:34.045633 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:34.045650 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:34.045655 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:34.045759 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:34.045857 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 03:06:34.045889 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 03:06:34.046254 18265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:34.039973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33675 (local address 127.0.98.2:44863)
0924 03:06:34.040163 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:34.040211 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:34.040749 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:34.041232 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:34.041237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:34.041249 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:06:34.042094 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:34.042103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.042931 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.042932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:34.044199 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:34.044201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.044624 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.044625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:34.044777 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:34.045442 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:06:34.045442 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:06:34.045474 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:06:34.045504 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:06:34.045764 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:06:34.045771 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:06:34.045774 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:06:34.045858 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 03:06:34.045963 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
I20260924 03:06:34.046938 18263 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33675
I20260924 03:06:34.047052 18263 heartbeater.cc:464] Registering TS with master...
I20260924 03:06:34.047263 18263 heartbeater.cc:511] Master 127.0.98.62:33675 requested a full tablet report, sending...
I20260924 03:06:34.047698 17944 ts_manager.cc:194] Registered new tserver with Master: 73e0fa2aa8654234a84a74878a3cd4db (127.0.98.2:45339)
I20260924 03:06:34.048075 17944 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:44863
W20260924 03:06:34.124786 18266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:34.124985 18266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:34.125008 18266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:34.126197 18266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:34.126240 18266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:34.126266 18266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:06:34.127723 18266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:34.127964 18266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:34.128294 18266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:34.133598 18274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:34.133562 18271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:34.133574 18272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:34.133800 18266 server_base.cc:1034] running on GCE node
I20260924 03:06:34.134060 18266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:34.134328 18266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:34.135545 18266 hybrid_clock.cc:648] HybridClock initialized: now 1790219194135541 us; error 40 us; skew 500 ppm
I20260924 03:06:34.136976 18266 webserver.cc:460] Webserver started at http://127.0.98.3:41537/ using document root <none> and password file <none>
I20260924 03:06:34.137277 18266 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:34.137363 18266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:34.137521 18266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:34.138772 18266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-2/data/instance:
uuid: "a02cc2135b71471aaca8eda99bb02ed5"
format_stamp: "Formatted at 2026-09-24 03:06:34 on dist-test-slave-wgsb"
server_key: "9d8e989235deace383660de3d5d52d2c"
server_key_iv: "4fa367021ab534f1c944a3185968b50f"
server_key_version: "encryptionkey@0"
I20260924 03:06:34.139185 18266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a02cc2135b71471aaca8eda99bb02ed5"
format_stamp: "Formatted at 2026-09-24 03:06:34 on dist-test-slave-wgsb"
server_key: "9d8e989235deace383660de3d5d52d2c"
server_key_iv: "4fa367021ab534f1c944a3185968b50f"
server_key_version: "encryptionkey@0"
I20260924 03:06:34.140713 18266 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:06:34.141552 18280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:34.141743 18266 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:06:34.141848 18266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "a02cc2135b71471aaca8eda99bb02ed5"
format_stamp: "Formatted at 2026-09-24 03:06:34 on dist-test-slave-wgsb"
server_key: "9d8e989235deace383660de3d5d52d2c"
server_key_iv: "4fa367021ab534f1c944a3185968b50f"
server_key_version: "encryptionkey@0"
I20260924 03:06:34.141927 18266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:34.159466 18266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:34.159853 18266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:34.160022 18266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:34.160279 18266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:06:34.160714 18266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:06:34.160778 18266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:34.160826 18266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:06:34.160856 18266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:34.168509 18266 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36017
I20260924 03:06:34.168550 18393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36017 every 8 connection(s)
I20260924 03:06:34.168937 18266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:06:34.174446 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:34.169769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48333 (local address 127.0.98.62:33675)
0924 03:06:34.169823 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:34.169825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:34.170081 (+ 256us) server_negotiation.cc:407] Connection header received
0924 03:06:34.170603 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:34.170604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:34.170624 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:34.170677 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:34.171373 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.171999 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.173033 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.173230 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.173291 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:34.174177 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:34.174178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:34.174179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:34.174183 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:34.174198 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:34.174203 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:34.174259 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:34.174355 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 03:06:34.174382 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 03:06:34.174639 18396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:34.169752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33675 (local address 127.0.98.3:48333)
0924 03:06:34.169946 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:34.169979 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:34.170478 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:34.170739 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:34.170742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:34.170750 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:06:34.171265 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:34.171269 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.172108 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.172109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:34.172933 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:34.172935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.173310 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.173311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:34.173441 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:34.174024 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:06:34.174024 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:06:34.174052 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:06:34.174076 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:06:34.174280 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:06:34.174283 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:06:34.174286 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:06:34.174338 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 03:06:34.174414 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
I20260924 03:06:34.175179 18394 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33675
I20260924 03:06:34.175287 18394 heartbeater.cc:464] Registering TS with master...
I20260924 03:06:34.175493 18394 heartbeater.cc:511] Master 127.0.98.62:33675 requested a full tablet report, sending...
I20260924 03:06:34.175958 17944 ts_manager.cc:194] Registered new tserver with Master: a02cc2135b71471aaca8eda99bb02ed5 (127.0.98.3:36017)
I20260924 03:06:34.176388 17944 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:48333
I20260924 03:06:34.177839 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 18266
I20260924 03:06:34.177925 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:06:34.178092 392 external_mini_cluster.cc:1428] Setting key b7a4b2b81ff486c9a94c27c9ffff0706
I20260924 03:06:34.179296 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:06:34.183825 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:34.180733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34284 (local address 127.0.98.62:33675)
0924 03:06:34.180816 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:34.180818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:34.180917 (+ 99us) server_negotiation.cc:407] Connection header received
0924 03:06:34.180997 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:34.180998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:34.181013 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:34.181076 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:34.181317 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.181792 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.182559 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.182766 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.182849 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:34.183548 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:34.183549 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:34.183550 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:34.183554 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:34.183568 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:34.183572 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:34.183657 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:34.183715 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 03:06:34.183739 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 03:06:34.185530 17944 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:06:34.187805 17944 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:06:34.194979 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:34.190323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42473 (local address 127.0.0.1:55242)
0924 03:06:34.190530 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:34.190572 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:34.190648 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:34.191330 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:34.191330 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:34.191332 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:34.191367 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:34.191368 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.192351 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.192352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:34.193600 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:34.193603 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.194633 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.194635 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:34.194722 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:34.194738 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 03:06:34.194826 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 03:06:34.194883 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":93,"threads_started":1}
I20260924 03:06:34.194981 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:34.190826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55242 (local address 127.0.98.1:42473)
0924 03:06:34.191134 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:34.191138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:34.191149 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:06:34.191189 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:34.191191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:34.191208 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:34.191327 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:34.191435 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.192251 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.193731 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.194524 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.194614 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:34.194709 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:34.194844 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 03:06:34.194888 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":90,"threads_started":1}
I20260924 03:06:34.195518 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:34.190664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45339 (local address 127.0.0.1:54622)
0924 03:06:34.190797 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:34.190807 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:34.190852 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:34.191228 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:34.191232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:34.191234 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:34.191339 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:34.191341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.192858 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.192859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:34.194047 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:34.194049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.195164 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.195165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:34.195338 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:34.195344 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 03:06:34.195404 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 03:06:34.195445 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":62,"thread_start_us":88,"threads_started":1}
I20260924 03:06:34.195844 18406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:34.190767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54622 (local address 127.0.98.2:45339)
0924 03:06:34.190973 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:34.190978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:34.190988 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:06:34.191031 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:34.191035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:34.191052 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:34.191137 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:34.191438 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.192768 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.194140 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.194755 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.194835 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:34.194917 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:34.195707 (+ 790us) server_negotiation.cc:299] Negotiation successful
0924 03:06:34.195754 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":109,"threads_started":1}
I20260924 03:06:34.195897 18066 tablet_service.cc:1467] Processing CreateTablet for tablet 7cbb21f1c7ed48ecb71a21a374470619 (DEFAULT_TABLE table=test-table [id=4584dda46b814c57b4a9e2c800a5dd63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:06:34.196324 18066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cbb21f1c7ed48ecb71a21a374470619. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:34.196501 18197 tablet_service.cc:1467] Processing CreateTablet for tablet 7cbb21f1c7ed48ecb71a21a374470619 (DEFAULT_TABLE table=test-table [id=4584dda46b814c57b4a9e2c800a5dd63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:06:34.196851 18197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cbb21f1c7ed48ecb71a21a374470619. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:34.199447 18408 tablet_bootstrap.cc:492] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f: Bootstrap starting.
I20260924 03:06:34.199944 18404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:34.190537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55544 (local address 127.0.98.3:36017)
0924 03:06:34.190737 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:34.190741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:34.190753 (+ 12us) server_negotiation.cc:407] Connection header received
0924 03:06:34.191188 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:34.191191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:34.191216 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:34.191278 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:34.191508 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.192758 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.194219 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.194992 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.195056 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:34.199508 (+ 4452us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:34.199783 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 03:06:34.199842 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":92,"threads_started":1}
I20260924 03:06:34.199970 18409 tablet_bootstrap.cc:492] T 7cbb21f1c7ed48ecb71a21a374470619 P 73e0fa2aa8654234a84a74878a3cd4db: Bootstrap starting.
I20260924 03:06:34.200243 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:34.190393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36017 (local address 127.0.0.1:55544)
0924 03:06:34.190563 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:34.190574 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:34.191015 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:34.191376 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:34.191377 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:34.191380 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:34.191431 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:34.191433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.192869 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.192870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:34.194108 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:34.194111 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.199579 (+ 5468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.199580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:34.199674 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:34.199678 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:06:34.200114 (+ 436us) client_negotiation.cc:240] Negotiation successful
0924 03:06:34.200167 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":86,"thread_start_us":115,"threads_started":1}
I20260924 03:06:34.200306 18408 tablet_bootstrap.cc:654] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:34.200637 18409 tablet_bootstrap.cc:654] T 7cbb21f1c7ed48ecb71a21a374470619 P 73e0fa2aa8654234a84a74878a3cd4db: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:34.200655 18408 log.cc:824] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:34.200975 18409 log.cc:824] T 7cbb21f1c7ed48ecb71a21a374470619 P 73e0fa2aa8654234a84a74878a3cd4db: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:34.201045 18328 tablet_service.cc:1467] Processing CreateTablet for tablet 7cbb21f1c7ed48ecb71a21a374470619 (DEFAULT_TABLE table=test-table [id=4584dda46b814c57b4a9e2c800a5dd63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:06:34.201459 18328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cbb21f1c7ed48ecb71a21a374470619. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:34.201651 18408 tablet_bootstrap.cc:492] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f: No bootstrap required, opened a new log
I20260924 03:06:34.201737 18408 ts_tablet_manager.cc:1397] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:06:34.201870 18409 tablet_bootstrap.cc:492] T 7cbb21f1c7ed48ecb71a21a374470619 P 73e0fa2aa8654234a84a74878a3cd4db: No bootstrap required, opened a new log
I20260924 03:06:34.201993 18409 ts_tablet_manager.cc:1397] T 7cbb21f1c7ed48ecb71a21a374470619 P 73e0fa2aa8654234a84a74878a3cd4db: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:06:34.203994 18408 raft_consensus.cc:348] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a27c7844ab549e588dbaec4b0fa780f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "a02cc2135b71471aaca8eda99bb02ed5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } peers { permanent_uuid: "73e0fa2aa8654234a84a74878a3cd4db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45339 } }
I20260924 03:06:34.204133 18408 raft_consensus.cc:374] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:34.204157 18408 raft_consensus.cc:729] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a27c7844ab549e588dbaec4b0fa780f, State: Initialized, Role: FOLLOWER
I20260924 03:06:34.204282 18408 consensus_queue.cc:260] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a27c7844ab549e588dbaec4b0fa780f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "a02cc2135b71471aaca8eda99bb02ed5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } peers { permanent_uuid: "73e0fa2aa8654234a84a74878a3cd4db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45339 } }
I20260924 03:06:34.204553 18408 ts_tablet_manager.cc:1428] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:06:34.204576 18132 heartbeater.cc:503] Master 127.0.98.62:33675 was elected leader, sending a full tablet report...
I20260924 03:06:34.205109 18413 tablet_bootstrap.cc:492] T 7cbb21f1c7ed48ecb71a21a374470619 P a02cc2135b71471aaca8eda99bb02ed5: Bootstrap starting.
I20260924 03:06:34.205550 18409 raft_consensus.cc:348] T 7cbb21f1c7ed48ecb71a21a374470619 P 73e0fa2aa8654234a84a74878a3cd4db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a27c7844ab549e588dbaec4b0fa780f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "a02cc2135b71471aaca8eda99bb02ed5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } peers { permanent_uuid: "73e0fa2aa8654234a84a74878a3cd4db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45339 } }
I20260924 03:06:34.205781 18409 raft_consensus.cc:374] T 7cbb21f1c7ed48ecb71a21a374470619 P 73e0fa2aa8654234a84a74878a3cd4db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:34.205832 18409 raft_consensus.cc:729] T 7cbb21f1c7ed48ecb71a21a374470619 P 73e0fa2aa8654234a84a74878a3cd4db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73e0fa2aa8654234a84a74878a3cd4db, State: Initialized, Role: FOLLOWER
I20260924 03:06:34.205843 18413 tablet_bootstrap.cc:654] T 7cbb21f1c7ed48ecb71a21a374470619 P a02cc2135b71471aaca8eda99bb02ed5: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:34.205983 18409 consensus_queue.cc:260] T 7cbb21f1c7ed48ecb71a21a374470619 P 73e0fa2aa8654234a84a74878a3cd4db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a27c7844ab549e588dbaec4b0fa780f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "a02cc2135b71471aaca8eda99bb02ed5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } peers { permanent_uuid: "73e0fa2aa8654234a84a74878a3cd4db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45339 } }
I20260924 03:06:34.206210 18413 log.cc:824] T 7cbb21f1c7ed48ecb71a21a374470619 P a02cc2135b71471aaca8eda99bb02ed5: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:34.206271 18409 ts_tablet_manager.cc:1428] T 7cbb21f1c7ed48ecb71a21a374470619 P 73e0fa2aa8654234a84a74878a3cd4db: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 03:06:34.206346 18263 heartbeater.cc:503] Master 127.0.98.62:33675 was elected leader, sending a full tablet report...
I20260924 03:06:34.207042 18413 tablet_bootstrap.cc:492] T 7cbb21f1c7ed48ecb71a21a374470619 P a02cc2135b71471aaca8eda99bb02ed5: No bootstrap required, opened a new log
I20260924 03:06:34.207119 18413 ts_tablet_manager.cc:1397] T 7cbb21f1c7ed48ecb71a21a374470619 P a02cc2135b71471aaca8eda99bb02ed5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:06:34.209110 18413 raft_consensus.cc:348] T 7cbb21f1c7ed48ecb71a21a374470619 P a02cc2135b71471aaca8eda99bb02ed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a27c7844ab549e588dbaec4b0fa780f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "a02cc2135b71471aaca8eda99bb02ed5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } peers { permanent_uuid: "73e0fa2aa8654234a84a74878a3cd4db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45339 } }
I20260924 03:06:34.209231 18413 raft_consensus.cc:374] T 7cbb21f1c7ed48ecb71a21a374470619 P a02cc2135b71471aaca8eda99bb02ed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:34.209254 18413 raft_consensus.cc:729] T 7cbb21f1c7ed48ecb71a21a374470619 P a02cc2135b71471aaca8eda99bb02ed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a02cc2135b71471aaca8eda99bb02ed5, State: Initialized, Role: FOLLOWER
I20260924 03:06:34.209393 18413 consensus_queue.cc:260] T 7cbb21f1c7ed48ecb71a21a374470619 P a02cc2135b71471aaca8eda99bb02ed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a27c7844ab549e588dbaec4b0fa780f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "a02cc2135b71471aaca8eda99bb02ed5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } peers { permanent_uuid: "73e0fa2aa8654234a84a74878a3cd4db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45339 } }
I20260924 03:06:34.209666 18413 ts_tablet_manager.cc:1428] T 7cbb21f1c7ed48ecb71a21a374470619 P a02cc2135b71471aaca8eda99bb02ed5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:06:34.209733 18394 heartbeater.cc:503] Master 127.0.98.62:33675 was elected leader, sending a full tablet report...
W20260924 03:06:34.289784 18264 tablet.cc:2369] T 7cbb21f1c7ed48ecb71a21a374470619 P 73e0fa2aa8654234a84a74878a3cd4db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:06:34.298902 18412 raft_consensus.cc:482] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:06:34.299038 18412 raft_consensus.cc:504] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a27c7844ab549e588dbaec4b0fa780f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "a02cc2135b71471aaca8eda99bb02ed5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } peers { permanent_uuid: "73e0fa2aa8654234a84a74878a3cd4db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45339 } }
I20260924 03:06:34.299356 18412 leader_election.cc:288] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a02cc2135b71471aaca8eda99bb02ed5 (127.0.98.3:36017), 73e0fa2aa8654234a84a74878a3cd4db (127.0.98.2:45339)
I20260924 03:06:34.302374 18404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:34.299648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45307 (local address 127.0.98.3:36017)
0924 03:06:34.299694 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:34.299696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:34.299728 (+ 32us) server_negotiation.cc:407] Connection header received
0924 03:06:34.299774 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:34.299775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:34.299785 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:34.299858 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:34.300059 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.300538 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.301567 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.302041 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.302095 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:34.302103 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:34.302292 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 03:06:34.302320 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 03:06:34.302392 18134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:34.299558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36017 (local address 127.0.98.1:45307)
0924 03:06:34.299652 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:34.299666 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:34.299718 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:34.299871 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:34.299872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:34.299874 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:34.299951 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:34.299953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.300614 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.300615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:34.301472 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:34.301474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.302143 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.302144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:34.302214 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:34.302218 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:06:34.302275 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 03:06:34.302314 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 03:06:34.302747 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:34.299731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45339 (local address 127.0.98.1:45501)
0924 03:06:34.299897 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:34.299906 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:34.299997 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:34.300125 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:34.300125 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:34.300127 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:34.300199 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:34.300200 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.300945 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.300946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:34.301818 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:34.301820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.302497 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.302498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:34.302583 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:34.302586 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:06:34.302632 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 03:06:34.302666 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":105,"threads_started":1}
I20260924 03:06:34.302785 18348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cbb21f1c7ed48ecb71a21a374470619" candidate_uuid: "0a27c7844ab549e588dbaec4b0fa780f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a02cc2135b71471aaca8eda99bb02ed5" is_pre_election: true
I20260924 03:06:34.302955 18348 raft_consensus.cc:2454] T 7cbb21f1c7ed48ecb71a21a374470619 P a02cc2135b71471aaca8eda99bb02ed5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a27c7844ab549e588dbaec4b0fa780f in term 0.
I20260924 03:06:34.302964 18406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:34.299826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45501 (local address 127.0.98.2:45339)
0924 03:06:34.299874 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:34.299876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:34.299972 (+ 96us) server_negotiation.cc:407] Connection header received
0924 03:06:34.300062 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:34.300063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:34.300072 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:34.300117 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:34.300277 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.300842 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.301907 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.302393 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.302455 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:34.302463 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:34.302855 (+ 392us) server_negotiation.cc:299] Negotiation successful
0924 03:06:34.302902 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 03:06:34.303222 18021 leader_election.cc:302] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a27c7844ab549e588dbaec4b0fa780f, a02cc2135b71471aaca8eda99bb02ed5; no voters:
I20260924 03:06:34.303229 18217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cbb21f1c7ed48ecb71a21a374470619" candidate_uuid: "0a27c7844ab549e588dbaec4b0fa780f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73e0fa2aa8654234a84a74878a3cd4db" is_pre_election: true
I20260924 03:06:34.303362 18217 raft_consensus.cc:2454] T 7cbb21f1c7ed48ecb71a21a374470619 P 73e0fa2aa8654234a84a74878a3cd4db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a27c7844ab549e588dbaec4b0fa780f in term 0.
I20260924 03:06:34.303355 18412 raft_consensus.cc:2781] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:06:34.303406 18412 raft_consensus.cc:482] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:06:34.303445 18412 raft_consensus.cc:3037] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:34.304241 18412 raft_consensus.cc:504] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a27c7844ab549e588dbaec4b0fa780f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "a02cc2135b71471aaca8eda99bb02ed5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } peers { permanent_uuid: "73e0fa2aa8654234a84a74878a3cd4db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45339 } }
I20260924 03:06:34.304373 18412 leader_election.cc:288] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f [CANDIDATE]: Term 1 election: Requested vote from peers a02cc2135b71471aaca8eda99bb02ed5 (127.0.98.3:36017), 73e0fa2aa8654234a84a74878a3cd4db (127.0.98.2:45339)
I20260924 03:06:34.304509 18348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cbb21f1c7ed48ecb71a21a374470619" candidate_uuid: "0a27c7844ab549e588dbaec4b0fa780f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a02cc2135b71471aaca8eda99bb02ed5"
I20260924 03:06:34.304530 18217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cbb21f1c7ed48ecb71a21a374470619" candidate_uuid: "0a27c7844ab549e588dbaec4b0fa780f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73e0fa2aa8654234a84a74878a3cd4db"
I20260924 03:06:34.304596 18348 raft_consensus.cc:3037] T 7cbb21f1c7ed48ecb71a21a374470619 P a02cc2135b71471aaca8eda99bb02ed5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:34.304596 18217 raft_consensus.cc:3037] T 7cbb21f1c7ed48ecb71a21a374470619 P 73e0fa2aa8654234a84a74878a3cd4db [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:34.305543 18217 raft_consensus.cc:2454] T 7cbb21f1c7ed48ecb71a21a374470619 P 73e0fa2aa8654234a84a74878a3cd4db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a27c7844ab549e588dbaec4b0fa780f in term 1.
I20260924 03:06:34.305544 18348 raft_consensus.cc:2454] T 7cbb21f1c7ed48ecb71a21a374470619 P a02cc2135b71471aaca8eda99bb02ed5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a27c7844ab549e588dbaec4b0fa780f in term 1.
I20260924 03:06:34.305742 18021 leader_election.cc:302] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a27c7844ab549e588dbaec4b0fa780f, 73e0fa2aa8654234a84a74878a3cd4db; no voters:
I20260924 03:06:34.305823 18412 raft_consensus.cc:2781] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:06:34.305985 18412 raft_consensus.cc:686] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f [term 1 LEADER]: Becoming Leader. State: Replica: 0a27c7844ab549e588dbaec4b0fa780f, State: Running, Role: LEADER
I20260924 03:06:34.306139 18412 consensus_queue.cc:237] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a27c7844ab549e588dbaec4b0fa780f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "a02cc2135b71471aaca8eda99bb02ed5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } peers { permanent_uuid: "73e0fa2aa8654234a84a74878a3cd4db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45339 } }
I20260924 03:06:34.306921 17944 catalog_manager.cc:5495] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a27c7844ab549e588dbaec4b0fa780f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0a27c7844ab549e588dbaec4b0fa780f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a27c7844ab549e588dbaec4b0fa780f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a02cc2135b71471aaca8eda99bb02ed5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73e0fa2aa8654234a84a74878a3cd4db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45339 } health_report { overall_health: UNKNOWN } } }
I20260924 03:06:34.339720 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:34.337196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55252 (local address 127.0.98.1:42473)
0924 03:06:34.337281 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:34.337283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:34.337293 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:06:34.337334 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:34.337335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:34.337360 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:34.337422 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:06:34.337665 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.338147 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.338971 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:34.339166 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:34.339224 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:34.339379 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:06:34.339478 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:06:34.339544 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:34.339622 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 03:06:34.339641 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 03:06:34.341733 18217 raft_consensus.cc:1261] T 7cbb21f1c7ed48ecb71a21a374470619 P 73e0fa2aa8654234a84a74878a3cd4db [term 1 FOLLOWER]: Refusing update from remote peer 0a27c7844ab549e588dbaec4b0fa780f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:06:34.341763 18348 raft_consensus.cc:1261] T 7cbb21f1c7ed48ecb71a21a374470619 P a02cc2135b71471aaca8eda99bb02ed5 [term 1 FOLLOWER]: Refusing update from remote peer 0a27c7844ab549e588dbaec4b0fa780f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:06:34.342159 18412 consensus_queue.cc:1035] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a02cc2135b71471aaca8eda99bb02ed5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:06:34.342344 18418 consensus_queue.cc:1035] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f [LEADER]: Connected to new peer: Peer: permanent_uuid: "73e0fa2aa8654234a84a74878a3cd4db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:06:34.344837 18423 mvcc.cc:205] Tried to move back new op lower bound from 7332737820020805632 to 7332737819878498304. Current Snapshot: MvccSnapshot[applied={T|T < 7332737820020805632}]
I20260924 03:06:34.347324 17943 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:06:34.347424 17943 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:06:34.348369 17943 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 281f4d58a73d4931be3a572428ccd254: Sending DeleteTablet for 3 replicas of tablet 7cbb21f1c7ed48ecb71a21a374470619
I20260924 03:06:34.348719 18066 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cbb21f1c7ed48ecb71a21a374470619 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:06:34 UTC) from {username='slave'} at 127.0.0.1:55242
I20260924 03:06:34.348757 18328 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cbb21f1c7ed48ecb71a21a374470619 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:06:34 UTC) from {username='slave'} at 127.0.0.1:55544
I20260924 03:06:34.348769 18197 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cbb21f1c7ed48ecb71a21a374470619 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:06:34 UTC) from {username='slave'} at 127.0.0.1:54622
I20260924 03:06:34.348984 18432 tablet_replica.cc:330] T 7cbb21f1c7ed48ecb71a21a374470619 P a02cc2135b71471aaca8eda99bb02ed5: stopping tablet replica
I20260924 03:06:34.348996 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 18004
I20260924 03:06:34.349128 18431 tablet_replica.cc:330] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f: stopping tablet replica
I20260924 03:06:34.349252 18431 raft_consensus.cc:2229] T 7cbb21f1c7ed48ecb71a21a374470619 P 0a27c7844ab549e588dbaec4b0fa780f [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:06:34.349598 18432 raft_consensus.cc:2229] T 7cbb21f1c7ed48ecb71a21a374470619 P a02cc2135b71471aaca8eda99bb02ed5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:06:34.349722 18432 raft_consensus.cc:2258] T 7cbb21f1c7ed48ecb71a21a374470619 P a02cc2135b71471aaca8eda99bb02ed5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:06:34.349877 18433 tablet_replica.cc:330] T 7cbb21f1c7ed48ecb71a21a374470619 P 73e0fa2aa8654234a84a74878a3cd4db: stopping tablet replica
I20260924 03:06:34.349973 18433 raft_consensus.cc:2229] T 7cbb21f1c7ed48ecb71a21a374470619 P 73e0fa2aa8654234a84a74878a3cd4db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:06:34.350080 18433 raft_consensus.cc:2258] T 7cbb21f1c7ed48ecb71a21a374470619 P 73e0fa2aa8654234a84a74878a3cd4db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:06:34.350932 18432 ts_tablet_manager.cc:1905] T 7cbb21f1c7ed48ecb71a21a374470619 P a02cc2135b71471aaca8eda99bb02ed5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:06:34.353030 18432 ts_tablet_manager.cc:1918] T 7cbb21f1c7ed48ecb71a21a374470619 P a02cc2135b71471aaca8eda99bb02ed5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:06:34.353118 18432 log.cc:1196] T 7cbb21f1c7ed48ecb71a21a374470619 P a02cc2135b71471aaca8eda99bb02ed5: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-2/wal/wals/7cbb21f1c7ed48ecb71a21a374470619
I20260924 03:06:34.353360 18432 ts_tablet_manager.cc:1939] T 7cbb21f1c7ed48ecb71a21a374470619 P a02cc2135b71471aaca8eda99bb02ed5: Deleting consensus metadata
I20260924 03:06:34.353790 17930 catalog_manager.cc:4841] TS a02cc2135b71471aaca8eda99bb02ed5 (127.0.98.3:36017): tablet 7cbb21f1c7ed48ecb71a21a374470619 (table test-table [id=4584dda46b814c57b4a9e2c800a5dd63]) successfully deleted
I20260924 03:06:34.353897 18433 ts_tablet_manager.cc:1905] T 7cbb21f1c7ed48ecb71a21a374470619 P 73e0fa2aa8654234a84a74878a3cd4db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:06:34.355767 18433 ts_tablet_manager.cc:1918] T 7cbb21f1c7ed48ecb71a21a374470619 P 73e0fa2aa8654234a84a74878a3cd4db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:06:34.355845 18433 log.cc:1196] T 7cbb21f1c7ed48ecb71a21a374470619 P 73e0fa2aa8654234a84a74878a3cd4db: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218948286999-392-0/minicluster-data/ts-1/wal/wals/7cbb21f1c7ed48ecb71a21a374470619
I20260924 03:06:34.356094 18433 ts_tablet_manager.cc:1939] T 7cbb21f1c7ed48ecb71a21a374470619 P 73e0fa2aa8654234a84a74878a3cd4db: Deleting consensus metadata
W20260924 03:06:34.356387 17929 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42473 (error 108)
I20260924 03:06:34.356803 17930 catalog_manager.cc:4841] TS 73e0fa2aa8654234a84a74878a3cd4db (127.0.98.2:45339): tablet 7cbb21f1c7ed48ecb71a21a374470619 (table test-table [id=4584dda46b814c57b4a9e2c800a5dd63]) successfully deleted
I20260924 03:06:34.356922 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:34.356774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42473 (local address 127.0.0.1:55260)
0924 03:06:34.356823 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:34.356885 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 03:06:34.356946 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 18135
W20260924 03:06:34.357163 17929 catalog_manager.cc:4584] TS 0a27c7844ab549e588dbaec4b0fa780f (127.0.98.1:42473): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7cbb21f1c7ed48ecb71a21a374470619: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42473: connect: Connection refused (error 111)
I20260924 03:06:34.361896 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 18266
I20260924 03:06:34.366416 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 17913
2026-09-24T03:06:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (796 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:06:34 dist-test-slave-wgsb krb5kdc[18437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:06:34 dist-test-slave-wgsb krb5kdc[18437](info): set up 2 sockets
Sep 24 03:06:34 dist-test-slave-wgsb krb5kdc[18437](info): commencing operation
krb5kdc: starting...
W20260924 03:06:35.436012 392 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:06:35 dist-test-slave-wgsb krb5kdc[18437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:06:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:06:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:06:35.584899 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45679
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:06:35.716990 18450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:35.717303 18450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:35.717336 18450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:35.719506 18450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:06:35.719674 18450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:35.719712 18450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:35.719732 18450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:06:35.719749 18450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:06:35.722605 18450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:35.723031 18450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:35.723657 18450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:35.730567 18455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:35.730850 18456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:35.730887 18458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:35.731190 18450 server_base.cc:1034] running on GCE node
I20260924 03:06:35.731666 18450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:35.732425 18450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:35.733709 18450 hybrid_clock.cc:648] HybridClock initialized: now 1790219195733637 us; error 85 us; skew 500 ppm
Sep 24 03:06:35 dist-test-slave-wgsb krb5kdc[18437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:35.744397 18450 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:06:35.745069 18450 webserver.cc:460] Webserver started at http://127.0.98.62:40245/ using document root <none> and password file <none>
I20260924 03:06:35.745401 18450 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:35.745458 18450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:35.745604 18450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:35.746836 18450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "31df26482e1c45b19bd7a2bd9fb7023e"
format_stamp: "Formatted at 2026-09-24 03:06:35 on dist-test-slave-wgsb"
server_key: "6eddce85e5ceb19e2857f550e67c6903"
server_key_iv: "e834d19ef63d0e35cc84f85beafe6b35"
server_key_version: "encryptionkey@0"
I20260924 03:06:35.747233 18450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "31df26482e1c45b19bd7a2bd9fb7023e"
format_stamp: "Formatted at 2026-09-24 03:06:35 on dist-test-slave-wgsb"
server_key: "6eddce85e5ceb19e2857f550e67c6903"
server_key_iv: "e834d19ef63d0e35cc84f85beafe6b35"
server_key_version: "encryptionkey@0"
I20260924 03:06:35.748834 18450 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:06:35.749773 18465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:35.749995 18450 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:06:35.750084 18450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "31df26482e1c45b19bd7a2bd9fb7023e"
format_stamp: "Formatted at 2026-09-24 03:06:35 on dist-test-slave-wgsb"
server_key: "6eddce85e5ceb19e2857f550e67c6903"
server_key_iv: "e834d19ef63d0e35cc84f85beafe6b35"
server_key_version: "encryptionkey@0"
I20260924 03:06:35.750174 18450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:35.767036 18450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:35.769544 18450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:35.769729 18450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:35.776175 18450 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45679
I20260924 03:06:35.776238 18517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45679 every 8 connection(s)
I20260924 03:06:35.776640 18450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:06:35.777593 18518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:35.780072 18518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31df26482e1c45b19bd7a2bd9fb7023e: Bootstrap starting.
I20260924 03:06:35.780633 18518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31df26482e1c45b19bd7a2bd9fb7023e: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:35.780869 18518 log.cc:824] T 00000000000000000000000000000000 P 31df26482e1c45b19bd7a2bd9fb7023e: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:35.781592 18518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31df26482e1c45b19bd7a2bd9fb7023e: No bootstrap required, opened a new log
I20260924 03:06:35.783188 18518 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31df26482e1c45b19bd7a2bd9fb7023e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31df26482e1c45b19bd7a2bd9fb7023e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45679 } }
I20260924 03:06:35.783353 18518 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31df26482e1c45b19bd7a2bd9fb7023e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:35.783377 18518 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31df26482e1c45b19bd7a2bd9fb7023e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31df26482e1c45b19bd7a2bd9fb7023e, State: Initialized, Role: FOLLOWER
I20260924 03:06:35.783528 18518 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31df26482e1c45b19bd7a2bd9fb7023e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31df26482e1c45b19bd7a2bd9fb7023e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45679 } }
I20260924 03:06:35.783674 18518 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31df26482e1c45b19bd7a2bd9fb7023e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:06:35.783700 18518 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31df26482e1c45b19bd7a2bd9fb7023e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:06:35.783733 18518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31df26482e1c45b19bd7a2bd9fb7023e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:35.784121 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 18450
I20260924 03:06:35.784389 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:06:35.784610 392 external_mini_cluster.cc:1428] Setting key 44f7e4afcfe49bb4027ddf7acc564329
I20260924 03:06:35.784651 18518 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31df26482e1c45b19bd7a2bd9fb7023e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31df26482e1c45b19bd7a2bd9fb7023e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45679 } }
I20260924 03:06:35.784811 18518 leader_election.cc:302] T 00000000000000000000000000000000 P 31df26482e1c45b19bd7a2bd9fb7023e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31df26482e1c45b19bd7a2bd9fb7023e; no voters:
I20260924 03:06:35.785218 18518 leader_election.cc:288] T 00000000000000000000000000000000 P 31df26482e1c45b19bd7a2bd9fb7023e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:06:35.785411 18524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31df26482e1c45b19bd7a2bd9fb7023e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:06:35.785514 18518 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31df26482e1c45b19bd7a2bd9fb7023e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:06:35.785594 18524 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31df26482e1c45b19bd7a2bd9fb7023e [term 1 LEADER]: Becoming Leader. State: Replica: 31df26482e1c45b19bd7a2bd9fb7023e, State: Running, Role: LEADER
I20260924 03:06:35.785693 18524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31df26482e1c45b19bd7a2bd9fb7023e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31df26482e1c45b19bd7a2bd9fb7023e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45679 } }
I20260924 03:06:35.786068 18524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31df26482e1c45b19bd7a2bd9fb7023e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31df26482e1c45b19bd7a2bd9fb7023e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31df26482e1c45b19bd7a2bd9fb7023e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45679 } } }
I20260924 03:06:35.786229 18524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31df26482e1c45b19bd7a2bd9fb7023e [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:35.786445 18524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31df26482e1c45b19bd7a2bd9fb7023e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31df26482e1c45b19bd7a2bd9fb7023e. Latest consensus state: current_term: 1 leader_uuid: "31df26482e1c45b19bd7a2bd9fb7023e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31df26482e1c45b19bd7a2bd9fb7023e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45679 } } }
I20260924 03:06:35.786546 18524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31df26482e1c45b19bd7a2bd9fb7023e [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:35.786937 18530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:06:35.787519 18530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:06:35.789364 18530 catalog_manager.cc:1347] Generated new cluster ID: ffb6d71cfd514f1ca984cae5c628480a
I20260924 03:06:35.789433 18530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:06:35 dist-test-slave-wgsb krb5kdc[18437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:06:35.795550 18530 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:06:35.796286 18530 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:06:35.797938 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:35.785198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56330 (local address 127.0.98.62:45679)
0924 03:06:35.785382 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:35.785392 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:35.785428 (+ 36us) server_negotiation.cc:407] Connection header received
0924 03:06:35.785556 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:35.785565 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:35.785723 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:35.785900 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:35.786928 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:35.788382 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:35.789628 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:35.790246 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:35.790365 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:35.793476 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:35.793482 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:35.793482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:06:35.793498 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:35.795884 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:35.796357 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:35.796361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:35.796361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:35.796478 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:35.796789 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:35.796790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:35.796791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:35.797102 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:35.797273 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:35.797398 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 03:06:35.797676 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":70,"threads_started":1}
I20260924 03:06:35.808228 18530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31df26482e1c45b19bd7a2bd9fb7023e: Generated new TSK 0
I20260924 03:06:35.808445 18530 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:06:35.848662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45679
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:06:35.948129 18544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:35.948462 18544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:35.948500 18544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:35.950150 18544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:35.950206 18544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:35.950240 18544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:06:35.952046 18544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:35.952337 18544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:35.952601 18544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:35.958421 18552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:35.958461 18550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:35.958465 18549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:35.958781 18544 server_base.cc:1034] running on GCE node
I20260924 03:06:35.959059 18544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:35.959348 18544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:35.960708 18544 hybrid_clock.cc:648] HybridClock initialized: now 1790219195960578 us; error 80 us; skew 500 ppm
Sep 24 03:06:35 dist-test-slave-wgsb krb5kdc[18437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:35.964191 18544 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:06:35.964824 18544 webserver.cc:460] Webserver started at http://127.0.98.1:40439/ using document root <none> and password file <none>
I20260924 03:06:35.965372 18544 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:35.965546 18544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:35.965766 18544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:35.967268 18544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-0/data/instance:
uuid: "1bab5dba75aa4175a3a3e9a22755b335"
format_stamp: "Formatted at 2026-09-24 03:06:35 on dist-test-slave-wgsb"
server_key: "ef0f340e57032169a188c3f14e0a11cc"
server_key_iv: "b7653b2df0c2f04b1c1bf52660e7447e"
server_key_version: "encryptionkey@0"
I20260924 03:06:35.967775 18544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1bab5dba75aa4175a3a3e9a22755b335"
format_stamp: "Formatted at 2026-09-24 03:06:35 on dist-test-slave-wgsb"
server_key: "ef0f340e57032169a188c3f14e0a11cc"
server_key_iv: "b7653b2df0c2f04b1c1bf52660e7447e"
server_key_version: "encryptionkey@0"
I20260924 03:06:35.969452 18544 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:06:35.970287 18559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:35.970479 18544 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:06:35.970582 18544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "1bab5dba75aa4175a3a3e9a22755b335"
format_stamp: "Formatted at 2026-09-24 03:06:35 on dist-test-slave-wgsb"
server_key: "ef0f340e57032169a188c3f14e0a11cc"
server_key_iv: "b7653b2df0c2f04b1c1bf52660e7447e"
server_key_version: "encryptionkey@0"
I20260924 03:06:35.970676 18544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:35.987311 18544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:35.991580 18544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:35.991858 18544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:35.992180 18544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:06:35.992563 18544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:06:35.992632 18544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:35.992671 18544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:06:35.992707 18544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:36.001477 18544 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40319
I20260924 03:06:36.001538 18672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40319 every 8 connection(s)
I20260924 03:06:36.001906 18544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:06:36.008846 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 18544
I20260924 03:06:36.008953 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:06:36.009168 392 external_mini_cluster.cc:1428] Setting key c5251e247d290b438ba2e9db64203be6
Sep 24 03:06:36 dist-test-slave-wgsb krb5kdc[18437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:06:36.013924 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:36.002841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51661 (local address 127.0.98.62:45679)
0924 03:06:36.002935 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:36.002938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:36.003129 (+ 191us) server_negotiation.cc:407] Connection header received
0924 03:06:36.004576 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:36.004578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:36.004593 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:36.004655 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:36.005388 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.005935 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.007002 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.007227 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.007317 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:36.010317 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:36.010319 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:36.010320 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:06:36.010329 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:36.012093 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:36.012689 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:36.012691 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:36.012691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:36.012753 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:36.013002 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:36.013002 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:36.013003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:36.013163 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:36.013319 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:36.013655 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 03:06:36.013773 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 03:06:36.014004 18675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:36.002796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45679 (local address 127.0.98.1:51661)
0924 03:06:36.002987 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:36.003037 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:36.004379 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:36.004736 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:36.004741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:36.005039 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:06:36.005280 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:36.005285 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.006042 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.006044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:36.006901 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:36.006904 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.007333 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.007334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:36.007474 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:36.008017 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:06:36.008026 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:06:36.010213 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:06:36.012191 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:06:36.012201 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:06:36.012204 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:06:36.012600 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:06:36.012823 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:06:36.012823 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:06:36.012824 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:06:36.012936 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:06:36.013347 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:06:36.013354 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:06:36.013498 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 03:06:36.013630 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 03:06:36.013779 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":90,"threads_started":1}
I20260924 03:06:36.014614 18673 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45679
I20260924 03:06:36.014729 18673 heartbeater.cc:464] Registering TS with master...
I20260924 03:06:36.014943 18673 heartbeater.cc:511] Master 127.0.98.62:45679 requested a full tablet report, sending...
I20260924 03:06:36.015568 18482 ts_manager.cc:194] Registered new tserver with Master: 1bab5dba75aa4175a3a3e9a22755b335 (127.0.98.1:40319)
I20260924 03:06:36.016332 18482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51661
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:06:36.044786 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45679
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:06:36.135456 18678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:36.135718 18678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:36.135746 18678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:36.136951 18678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:36.136996 18678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:36.137024 18678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:06:36.138511 18678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:36.138743 18678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:36.138927 18678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:36.144498 18686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:36.144518 18684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:36.144464 18683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:36.144745 18678 server_base.cc:1034] running on GCE node
I20260924 03:06:36.145211 18678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:36.145524 18678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:36.146765 18678 hybrid_clock.cc:648] HybridClock initialized: now 1790219196146739 us; error 48 us; skew 500 ppm
Sep 24 03:06:36 dist-test-slave-wgsb krb5kdc[18437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:36.149921 18678 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:06:36.150616 18678 webserver.cc:460] Webserver started at http://127.0.98.2:41315/ using document root <none> and password file <none>
I20260924 03:06:36.150902 18678 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:36.150982 18678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:36.151183 18678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:36.152612 18678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-1/data/instance:
uuid: "3e410daf14be4830bf72da9d32fd645a"
format_stamp: "Formatted at 2026-09-24 03:06:36 on dist-test-slave-wgsb"
server_key: "8a5c985815f472f59b416eb66ad0de54"
server_key_iv: "6f81096a0dab51ec54327623a825f73f"
server_key_version: "encryptionkey@0"
I20260924 03:06:36.152952 18678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3e410daf14be4830bf72da9d32fd645a"
format_stamp: "Formatted at 2026-09-24 03:06:36 on dist-test-slave-wgsb"
server_key: "8a5c985815f472f59b416eb66ad0de54"
server_key_iv: "6f81096a0dab51ec54327623a825f73f"
server_key_version: "encryptionkey@0"
I20260924 03:06:36.154558 18678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:06:36.155400 18693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:36.155630 18678 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:06:36.155755 18678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "3e410daf14be4830bf72da9d32fd645a"
format_stamp: "Formatted at 2026-09-24 03:06:36 on dist-test-slave-wgsb"
server_key: "8a5c985815f472f59b416eb66ad0de54"
server_key_iv: "6f81096a0dab51ec54327623a825f73f"
server_key_version: "encryptionkey@0"
I20260924 03:06:36.155856 18678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:36.167848 18678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:36.170622 18678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:36.170825 18678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:36.171106 18678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:06:36.171484 18678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:06:36.171550 18678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:36.171633 18678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:06:36.171728 18678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:36.179276 18678 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44097
I20260924 03:06:36.179315 18806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44097 every 8 connection(s)
I20260924 03:06:36.179776 18678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:06:36.180860 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 18678
I20260924 03:06:36.181005 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:06:36.181237 392 external_mini_cluster.cc:1428] Setting key a076b2723fde58dfb16b449c40faf47e
Sep 24 03:06:36 dist-test-slave-wgsb krb5kdc[18437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 03:06:36.192488 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:36.180778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57207 (local address 127.0.98.62:45679)
0924 03:06:36.180858 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:36.180861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:36.181089 (+ 228us) server_negotiation.cc:407] Connection header received
0924 03:06:36.182424 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:36.182425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:36.182443 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:36.182516 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:36.183220 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.184153 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.185459 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.185876 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.185999 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:36.188573 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:36.188575 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:36.188576 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:06:36.188587 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:36.190516 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:36.191172 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:36.191173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:36.191174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:36.191242 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:36.191690 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:36.191691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:36.191691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:36.191812 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:36.191869 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:36.192318 (+ 449us) server_negotiation.cc:299] Negotiation successful
0924 03:06:36.192409 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 03:06:36.192770 18809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:36.180749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45679 (local address 127.0.98.2:57207)
0924 03:06:36.180948 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:36.181002 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:36.182303 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:36.182595 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:36.182600 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:36.182896 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:06:36.183114 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:36.183118 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.184293 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.184294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:36.185377 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:36.185379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.186067 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.186068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:36.186276 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:36.186889 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:06:36.186897 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:06:36.188473 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:06:36.190670 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:06:36.190680 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:06:36.190682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:06:36.191065 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:06:36.191336 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:06:36.191337 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:06:36.191337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:06:36.191472 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:06:36.191878 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:06:36.191884 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:06:36.192031 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 03:06:36.192208 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 03:06:36.192432 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":121,"threads_started":1}
I20260924 03:06:36.193509 18807 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45679
I20260924 03:06:36.193630 18807 heartbeater.cc:464] Registering TS with master...
I20260924 03:06:36.193856 18807 heartbeater.cc:511] Master 127.0.98.62:45679 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 03:06:36.194339 18482 ts_manager.cc:194] Registered new tserver with Master: 3e410daf14be4830bf72da9d32fd645a (127.0.98.2:44097)
I20260924 03:06:36.194768 18482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57207
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:06:36.226727 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45679
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:06:36.318804 18812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:36.319020 18812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:36.319042 18812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:36.320148 18812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:36.320182 18812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:36.320214 18812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:06:36.321668 18812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:36.321897 18812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:36.322082 18812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:36.327283 18817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:36.327245 18818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:36.327306 18812 server_base.cc:1034] running on GCE node
W20260924 03:06:36.327245 18820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:36.327795 18812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:36.328100 18812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:36.329295 18812 hybrid_clock.cc:648] HybridClock initialized: now 1790219196329283 us; error 30 us; skew 500 ppm
Sep 24 03:06:36 dist-test-slave-wgsb krb5kdc[18437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:36.332017 18812 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:06:36.332525 18812 webserver.cc:460] Webserver started at http://127.0.98.3:42339/ using document root <none> and password file <none>
I20260924 03:06:36.332764 18812 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:36.332834 18812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:36.332988 18812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:36.334059 18812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-2/data/instance:
uuid: "147a0711727045b39e5147066ec23d85"
format_stamp: "Formatted at 2026-09-24 03:06:36 on dist-test-slave-wgsb"
server_key: "450b0b2d061df7aab66ecbb0c4eb6ed2"
server_key_iv: "ed912f6467c65b213ea4082b049bad2e"
server_key_version: "encryptionkey@0"
I20260924 03:06:36.334452 18812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-2/wal/instance:
uuid: "147a0711727045b39e5147066ec23d85"
format_stamp: "Formatted at 2026-09-24 03:06:36 on dist-test-slave-wgsb"
server_key: "450b0b2d061df7aab66ecbb0c4eb6ed2"
server_key_iv: "ed912f6467c65b213ea4082b049bad2e"
server_key_version: "encryptionkey@0"
I20260924 03:06:36.335980 18812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260924 03:06:36.336880 18827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:36.337296 18812 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:06:36.337425 18812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "147a0711727045b39e5147066ec23d85"
format_stamp: "Formatted at 2026-09-24 03:06:36 on dist-test-slave-wgsb"
server_key: "450b0b2d061df7aab66ecbb0c4eb6ed2"
server_key_iv: "ed912f6467c65b213ea4082b049bad2e"
server_key_version: "encryptionkey@0"
I20260924 03:06:36.337510 18812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:36.344646 18812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:36.346992 18812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:36.347157 18812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:36.347388 18812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:06:36.347755 18812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:06:36.347810 18812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:36.347852 18812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:06:36.347879 18812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:36.353925 18812 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35653
I20260924 03:06:36.353972 18940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35653 every 8 connection(s)
I20260924 03:06:36.354290 18812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 03:06:36 dist-test-slave-wgsb krb5kdc[18437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:06:36.362458 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 18812
I20260924 03:06:36.362567 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:06:36.362794 392 external_mini_cluster.cc:1428] Setting key 6f2121072c37dd809c44e19aeec144f8
I20260924 03:06:36.365127 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:36.355066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52355 (local address 127.0.98.62:45679)
0924 03:06:36.355121 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:36.355123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:36.355377 (+ 254us) server_negotiation.cc:407] Connection header received
0924 03:06:36.356415 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:36.356416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:36.356429 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:36.356473 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:36.357114 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.357619 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.358733 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.358952 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.359016 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:36.361648 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:36.361650 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:36.361650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:06:36.361658 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:36.363510 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:36.364007 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:36.364008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:36.364008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:36.364063 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:36.364367 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:36.364368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:36.364368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:36.364500 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:36.364577 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:36.364917 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 03:06:36.364993 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:06:36.365216 18943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:36.355019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45679 (local address 127.0.98.3:52355)
0924 03:06:36.355248 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:36.355287 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:36.356295 (+ 1008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:36.356547 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:36.356551 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:36.356814 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:06:36.357003 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:36.357007 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.357739 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.357740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:36.358635 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:36.358637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.359024 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.359025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:36.359147 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:36.359913 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:06:36.359923 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:06:36.361545 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:06:36.363588 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:06:36.363616 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:06:36.363618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:06:36.363947 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:06:36.364151 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:06:36.364152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:06:36.364152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:06:36.364256 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:06:36.364598 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:06:36.364605 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:06:36.364762 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 03:06:36.364867 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 03:06:36.365012 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":125,"threads_started":1}
I20260924 03:06:36.365859 18941 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45679
I20260924 03:06:36.365964 18941 heartbeater.cc:464] Registering TS with master...
I20260924 03:06:36.366125 18941 heartbeater.cc:511] Master 127.0.98.62:45679 requested a full tablet report, sending...
I20260924 03:06:36.366532 18482 ts_manager.cc:194] Registered new tserver with Master: 147a0711727045b39e5147066ec23d85 (127.0.98.3:35653)
I20260924 03:06:36.367170 18482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52355
I20260924 03:06:36.375497 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:06:36.384637 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:36.377009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56342 (local address 127.0.98.62:45679)
0924 03:06:36.377074 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:36.377078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:36.377263 (+ 185us) server_negotiation.cc:407] Connection header received
0924 03:06:36.377318 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:36.377319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:36.377329 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:36.377386 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:36.378041 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.378620 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.379440 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.379731 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.379796 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:36.381559 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:36.381560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:36.381561 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:06:36.381569 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:36.383151 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:36.383730 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:36.383731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:36.383731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:36.383798 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:36.384081 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:36.384082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:36.384083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:36.384279 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:36.384380 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:36.384490 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 03:06:36.384544 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 03:06:36.386718 18482 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:06:36.388549 18482 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:06:36.396557 18952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:36.391970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40319 (local address 127.0.0.1:35466)
0924 03:06:36.392297 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:36.392325 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:36.392399 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:36.392968 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:36.392968 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:36.392972 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:36.393027 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:36.393030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.394184 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.394185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:36.395429 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:36.395431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.396222 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.396223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:36.396311 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:36.396327 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 03:06:36.396396 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 03:06:36.396454 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":36,"thread_start_us":93,"threads_started":1}
I20260924 03:06:36.396653 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:36.392041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49086 (local address 127.0.98.2:44097)
0924 03:06:36.392506 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:36.392516 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:36.392524 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:06:36.392556 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:36.392558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:36.392569 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:36.392659 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:36.392865 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.394019 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.395511 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.396270 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.396348 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:36.396402 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:36.396478 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 03:06:36.396526 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":87,"threads_started":1}
I20260924 03:06:36.396917 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:36.392096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35466 (local address 127.0.98.1:40319)
0924 03:06:36.392803 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:36.392808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:36.392818 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:06:36.392855 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:36.392857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:36.392871 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:36.392944 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:36.393127 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.394052 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.395524 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.396121 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.396215 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:36.396280 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:36.396793 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 03:06:36.396837 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":74,"threads_started":1}
I20260924 03:06:36.396921 18957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:36.392285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37334 (local address 127.0.98.3:35653)
0924 03:06:36.392480 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:36.392485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:36.393167 (+ 682us) server_negotiation.cc:407] Connection header received
0924 03:06:36.393210 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:36.393224 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:36.393240 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:36.393310 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:36.393429 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.394450 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.395704 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.396462 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.396531 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:36.396580 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:36.396794 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 03:06:36.396837 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":92,"threads_started":1}
I20260924 03:06:36.396557 18953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:36.391925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44097 (local address 127.0.0.1:49086)
0924 03:06:36.392285 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:36.392326 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:36.392394 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:36.392664 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:36.392668 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:36.392670 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:36.392770 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:36.392772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.394136 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.394138 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:36.395418 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:36.395421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.396346 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.396346 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:36.396409 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:36.396413 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:06:36.396462 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 03:06:36.396492 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":59,"threads_started":1}
I20260924 03:06:36.397733 18607 tablet_service.cc:1467] Processing CreateTablet for tablet d21d5e6d72c24e73b296c859773ff5df (DEFAULT_TABLE table=test-table [id=635d334681e64851a33cd35a27d47755]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:06:36.397889 18741 tablet_service.cc:1467] Processing CreateTablet for tablet d21d5e6d72c24e73b296c859773ff5df (DEFAULT_TABLE table=test-table [id=635d334681e64851a33cd35a27d47755]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:06:36.398126 18607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d21d5e6d72c24e73b296c859773ff5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:36.398257 18741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d21d5e6d72c24e73b296c859773ff5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:36.401048 18958 tablet_bootstrap.cc:492] T d21d5e6d72c24e73b296c859773ff5df P 3e410daf14be4830bf72da9d32fd645a: Bootstrap starting.
I20260924 03:06:36.396854 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:36.392135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35653 (local address 127.0.0.1:37334)
0924 03:06:36.393046 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:36.393058 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:36.393106 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:36.393313 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:36.393314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:36.393316 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:36.393354 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:36.393356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.394562 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.394563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:36.395633 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:36.395635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.396616 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.396617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:36.396685 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:36.396690 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:06:36.396750 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 03:06:36.396789 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":3,"thread_start_us":82,"threads_started":1}
I20260924 03:06:36.401204 18959 tablet_bootstrap.cc:492] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335: Bootstrap starting.
I20260924 03:06:36.401829 18959 tablet_bootstrap.cc:654] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:36.401839 18958 tablet_bootstrap.cc:654] T d21d5e6d72c24e73b296c859773ff5df P 3e410daf14be4830bf72da9d32fd645a: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:36.402138 18875 tablet_service.cc:1467] Processing CreateTablet for tablet d21d5e6d72c24e73b296c859773ff5df (DEFAULT_TABLE table=test-table [id=635d334681e64851a33cd35a27d47755]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:06:36.402369 18959 log.cc:824] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:36.402432 18958 log.cc:824] T d21d5e6d72c24e73b296c859773ff5df P 3e410daf14be4830bf72da9d32fd645a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:36.402519 18875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d21d5e6d72c24e73b296c859773ff5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:36.403210 18958 tablet_bootstrap.cc:492] T d21d5e6d72c24e73b296c859773ff5df P 3e410daf14be4830bf72da9d32fd645a: No bootstrap required, opened a new log
I20260924 03:06:36.403283 18958 ts_tablet_manager.cc:1397] T d21d5e6d72c24e73b296c859773ff5df P 3e410daf14be4830bf72da9d32fd645a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:06:36.404201 18959 tablet_bootstrap.cc:492] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335: No bootstrap required, opened a new log
I20260924 03:06:36.404289 18959 ts_tablet_manager.cc:1397] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 03:06:36.405490 18958 raft_consensus.cc:348] T d21d5e6d72c24e73b296c859773ff5df P 3e410daf14be4830bf72da9d32fd645a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e410daf14be4830bf72da9d32fd645a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44097 } } peers { permanent_uuid: "147a0711727045b39e5147066ec23d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35653 } } peers { permanent_uuid: "1bab5dba75aa4175a3a3e9a22755b335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40319 } }
I20260924 03:06:36.405710 18958 raft_consensus.cc:374] T d21d5e6d72c24e73b296c859773ff5df P 3e410daf14be4830bf72da9d32fd645a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:36.405772 18958 raft_consensus.cc:729] T d21d5e6d72c24e73b296c859773ff5df P 3e410daf14be4830bf72da9d32fd645a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e410daf14be4830bf72da9d32fd645a, State: Initialized, Role: FOLLOWER
I20260924 03:06:36.405936 18958 consensus_queue.cc:260] T d21d5e6d72c24e73b296c859773ff5df P 3e410daf14be4830bf72da9d32fd645a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e410daf14be4830bf72da9d32fd645a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44097 } } peers { permanent_uuid: "147a0711727045b39e5147066ec23d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35653 } } peers { permanent_uuid: "1bab5dba75aa4175a3a3e9a22755b335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40319 } }
I20260924 03:06:36.405961 18959 raft_consensus.cc:348] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e410daf14be4830bf72da9d32fd645a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44097 } } peers { permanent_uuid: "147a0711727045b39e5147066ec23d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35653 } } peers { permanent_uuid: "1bab5dba75aa4175a3a3e9a22755b335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40319 } }
I20260924 03:06:36.406147 18959 raft_consensus.cc:374] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:36.406198 18959 raft_consensus.cc:729] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bab5dba75aa4175a3a3e9a22755b335, State: Initialized, Role: FOLLOWER
I20260924 03:06:36.406220 18958 ts_tablet_manager.cc:1428] T d21d5e6d72c24e73b296c859773ff5df P 3e410daf14be4830bf72da9d32fd645a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 03:06:36.406333 18959 consensus_queue.cc:260] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e410daf14be4830bf72da9d32fd645a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44097 } } peers { permanent_uuid: "147a0711727045b39e5147066ec23d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35653 } } peers { permanent_uuid: "1bab5dba75aa4175a3a3e9a22755b335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40319 } }
I20260924 03:06:36.406385 18807 heartbeater.cc:503] Master 127.0.98.62:45679 was elected leader, sending a full tablet report...
I20260924 03:06:36.406486 18962 tablet_bootstrap.cc:492] T d21d5e6d72c24e73b296c859773ff5df P 147a0711727045b39e5147066ec23d85: Bootstrap starting.
I20260924 03:06:36.406628 18959 ts_tablet_manager.cc:1428] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 03:06:36.406654 18673 heartbeater.cc:503] Master 127.0.98.62:45679 was elected leader, sending a full tablet report...
I20260924 03:06:36.407305 18962 tablet_bootstrap.cc:654] T d21d5e6d72c24e73b296c859773ff5df P 147a0711727045b39e5147066ec23d85: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:36.408034 18962 log.cc:824] T d21d5e6d72c24e73b296c859773ff5df P 147a0711727045b39e5147066ec23d85: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:36.408906 18962 tablet_bootstrap.cc:492] T d21d5e6d72c24e73b296c859773ff5df P 147a0711727045b39e5147066ec23d85: No bootstrap required, opened a new log
I20260924 03:06:36.408970 18962 ts_tablet_manager.cc:1397] T d21d5e6d72c24e73b296c859773ff5df P 147a0711727045b39e5147066ec23d85: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 03:06:36.410391 18962 raft_consensus.cc:348] T d21d5e6d72c24e73b296c859773ff5df P 147a0711727045b39e5147066ec23d85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e410daf14be4830bf72da9d32fd645a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44097 } } peers { permanent_uuid: "147a0711727045b39e5147066ec23d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35653 } } peers { permanent_uuid: "1bab5dba75aa4175a3a3e9a22755b335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40319 } }
I20260924 03:06:36.410537 18962 raft_consensus.cc:374] T d21d5e6d72c24e73b296c859773ff5df P 147a0711727045b39e5147066ec23d85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:36.410559 18962 raft_consensus.cc:729] T d21d5e6d72c24e73b296c859773ff5df P 147a0711727045b39e5147066ec23d85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 147a0711727045b39e5147066ec23d85, State: Initialized, Role: FOLLOWER
I20260924 03:06:36.410701 18962 consensus_queue.cc:260] T d21d5e6d72c24e73b296c859773ff5df P 147a0711727045b39e5147066ec23d85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e410daf14be4830bf72da9d32fd645a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44097 } } peers { permanent_uuid: "147a0711727045b39e5147066ec23d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35653 } } peers { permanent_uuid: "1bab5dba75aa4175a3a3e9a22755b335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40319 } }
I20260924 03:06:36.411000 18962 ts_tablet_manager.cc:1428] T d21d5e6d72c24e73b296c859773ff5df P 147a0711727045b39e5147066ec23d85: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 03:06:36.411031 18941 heartbeater.cc:503] Master 127.0.98.62:45679 was elected leader, sending a full tablet report...
W20260924 03:06:36.430478 18808 tablet.cc:2369] T d21d5e6d72c24e73b296c859773ff5df P 3e410daf14be4830bf72da9d32fd645a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:06:36.502799 18674 tablet.cc:2369] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:06:36.554924 18964 raft_consensus.cc:482] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:06:36.555038 18964 raft_consensus.cc:504] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e410daf14be4830bf72da9d32fd645a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44097 } } peers { permanent_uuid: "147a0711727045b39e5147066ec23d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35653 } } peers { permanent_uuid: "1bab5dba75aa4175a3a3e9a22755b335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40319 } }
I20260924 03:06:36.555384 18964 leader_election.cc:288] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e410daf14be4830bf72da9d32fd645a (127.0.98.2:44097), 147a0711727045b39e5147066ec23d85 (127.0.98.3:35653)
I20260924 03:06:36.558460 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:36.555719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42161 (local address 127.0.98.2:44097)
0924 03:06:36.555777 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:36.555779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:36.555809 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:06:36.555907 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:36.555908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:36.555921 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:36.556000 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:36.556173 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.556663 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.557693 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.558076 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.558171 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:36.558182 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:36.558344 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 03:06:36.558384 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 03:06:36.558480 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:36.555584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44097 (local address 127.0.98.1:42161)
0924 03:06:36.555744 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:36.555758 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:36.555828 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:36.556006 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:36.556008 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:36.556011 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:36.556089 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:36.556093 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.556752 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.556753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:36.557583 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:36.557586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.558176 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.558177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:36.558259 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:36.558263 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:06:36.558330 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 03:06:36.558388 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260924 03:06:36.558538 18957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:36.555972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33015 (local address 127.0.98.3:35653)
0924 03:06:36.556024 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:36.556026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:36.556033 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:06:36.556066 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:36.556067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:36.556074 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:36.556133 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:36.556317 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.556776 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.557764 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.558135 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.558223 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:36.558234 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:36.558424 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 03:06:36.558469 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 03:06:36.558612 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:36.555846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35653 (local address 127.0.98.1:33015)
0924 03:06:36.555932 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:36.555942 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:36.555980 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:36.556153 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:36.556153 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:36.556156 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:36.556242 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:36.556244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.556862 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.556863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:36.557661 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:36.557663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.558228 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.558229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:36.558306 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:36.558309 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:06:36.558413 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 03:06:36.558458 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":49,"threads_started":1}
I20260924 03:06:36.558982 18761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d21d5e6d72c24e73b296c859773ff5df" candidate_uuid: "1bab5dba75aa4175a3a3e9a22755b335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e410daf14be4830bf72da9d32fd645a" is_pre_election: true
I20260924 03:06:36.558980 18895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d21d5e6d72c24e73b296c859773ff5df" candidate_uuid: "1bab5dba75aa4175a3a3e9a22755b335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "147a0711727045b39e5147066ec23d85" is_pre_election: true
I20260924 03:06:36.559152 18895 raft_consensus.cc:2454] T d21d5e6d72c24e73b296c859773ff5df P 147a0711727045b39e5147066ec23d85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bab5dba75aa4175a3a3e9a22755b335 in term 0.
I20260924 03:06:36.559152 18761 raft_consensus.cc:2454] T d21d5e6d72c24e73b296c859773ff5df P 3e410daf14be4830bf72da9d32fd645a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bab5dba75aa4175a3a3e9a22755b335 in term 0.
I20260924 03:06:36.559453 18563 leader_election.cc:302] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bab5dba75aa4175a3a3e9a22755b335, 3e410daf14be4830bf72da9d32fd645a; no voters:
I20260924 03:06:36.559662 18964 raft_consensus.cc:2781] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:06:36.559769 18964 raft_consensus.cc:482] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:06:36.559814 18964 raft_consensus.cc:3037] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:36.560595 18964 raft_consensus.cc:504] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e410daf14be4830bf72da9d32fd645a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44097 } } peers { permanent_uuid: "147a0711727045b39e5147066ec23d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35653 } } peers { permanent_uuid: "1bab5dba75aa4175a3a3e9a22755b335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40319 } }
I20260924 03:06:36.560747 18964 leader_election.cc:288] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335 [CANDIDATE]: Term 1 election: Requested vote from peers 3e410daf14be4830bf72da9d32fd645a (127.0.98.2:44097), 147a0711727045b39e5147066ec23d85 (127.0.98.3:35653)
I20260924 03:06:36.560999 18761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d21d5e6d72c24e73b296c859773ff5df" candidate_uuid: "1bab5dba75aa4175a3a3e9a22755b335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e410daf14be4830bf72da9d32fd645a"
I20260924 03:06:36.561017 18895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d21d5e6d72c24e73b296c859773ff5df" candidate_uuid: "1bab5dba75aa4175a3a3e9a22755b335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "147a0711727045b39e5147066ec23d85"
I20260924 03:06:36.561100 18895 raft_consensus.cc:3037] T d21d5e6d72c24e73b296c859773ff5df P 147a0711727045b39e5147066ec23d85 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:36.561100 18761 raft_consensus.cc:3037] T d21d5e6d72c24e73b296c859773ff5df P 3e410daf14be4830bf72da9d32fd645a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:36.562011 18761 raft_consensus.cc:2454] T d21d5e6d72c24e73b296c859773ff5df P 3e410daf14be4830bf72da9d32fd645a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bab5dba75aa4175a3a3e9a22755b335 in term 1.
I20260924 03:06:36.562021 18895 raft_consensus.cc:2454] T d21d5e6d72c24e73b296c859773ff5df P 147a0711727045b39e5147066ec23d85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bab5dba75aa4175a3a3e9a22755b335 in term 1.
I20260924 03:06:36.562284 18563 leader_election.cc:302] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bab5dba75aa4175a3a3e9a22755b335, 3e410daf14be4830bf72da9d32fd645a; no voters:
I20260924 03:06:36.562393 18964 raft_consensus.cc:2781] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:06:36.562623 18964 raft_consensus.cc:686] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335 [term 1 LEADER]: Becoming Leader. State: Replica: 1bab5dba75aa4175a3a3e9a22755b335, State: Running, Role: LEADER
I20260924 03:06:36.562749 18964 consensus_queue.cc:237] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e410daf14be4830bf72da9d32fd645a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44097 } } peers { permanent_uuid: "147a0711727045b39e5147066ec23d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35653 } } peers { permanent_uuid: "1bab5dba75aa4175a3a3e9a22755b335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40319 } }
I20260924 03:06:36.563771 18482 catalog_manager.cc:5495] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bab5dba75aa4175a3a3e9a22755b335 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1bab5dba75aa4175a3a3e9a22755b335" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e410daf14be4830bf72da9d32fd645a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "147a0711727045b39e5147066ec23d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bab5dba75aa4175a3a3e9a22755b335" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40319 } health_report { overall_health: HEALTHY } } }
I20260924 03:06:36.584332 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:36.581654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35476 (local address 127.0.98.1:40319)
0924 03:06:36.581726 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:36.581728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:36.581739 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:06:36.581776 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:36.581777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:36.581798 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:36.581866 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:06:36.582087 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.582642 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.583448 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.583699 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.583755 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:36.583975 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:06:36.584067 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:06:36.584143 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:36.584255 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 03:06:36.584266 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 03:06:36.586845 18895 raft_consensus.cc:1261] T d21d5e6d72c24e73b296c859773ff5df P 147a0711727045b39e5147066ec23d85 [term 1 FOLLOWER]: Refusing update from remote peer 1bab5dba75aa4175a3a3e9a22755b335: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:06:36.586845 18761 raft_consensus.cc:1261] T d21d5e6d72c24e73b296c859773ff5df P 3e410daf14be4830bf72da9d32fd645a [term 1 FOLLOWER]: Refusing update from remote peer 1bab5dba75aa4175a3a3e9a22755b335: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:06:36.587415 18969 consensus_queue.cc:1035] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335 [LEADER]: Connected to new peer: Peer: permanent_uuid: "147a0711727045b39e5147066ec23d85" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:06:36.587576 18964 consensus_queue.cc:1035] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e410daf14be4830bf72da9d32fd645a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:06:36.590097 18972 mvcc.cc:205] Tried to move back new op lower bound from 7332737829215551488 to 7332737829121732608. Current Snapshot: MvccSnapshot[applied={T|T < 7332737829215551488}]
I20260924 03:06:36.590365 18973 mvcc.cc:205] Tried to move back new op lower bound from 7332737829215551488 to 7332737829121732608. Current Snapshot: MvccSnapshot[applied={T|T < 7332737829215551488}]
I20260924 03:06:36.591167 18974 mvcc.cc:205] Tried to move back new op lower bound from 7332737829215551488 to 7332737829121732608. Current Snapshot: MvccSnapshot[applied={T|T < 7332737829215551488}]
I20260924 03:06:36.592832 18481 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:06:36.592948 18481 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:06:36.593935 18481 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31df26482e1c45b19bd7a2bd9fb7023e: Sending DeleteTablet for 3 replicas of tablet d21d5e6d72c24e73b296c859773ff5df
I20260924 03:06:36.594312 18607 tablet_service.cc:1514] Processing DeleteTablet for tablet d21d5e6d72c24e73b296c859773ff5df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:06:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35466
I20260924 03:06:36.594440 18741 tablet_service.cc:1514] Processing DeleteTablet for tablet d21d5e6d72c24e73b296c859773ff5df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:06:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49086
I20260924 03:06:36.594473 18875 tablet_service.cc:1514] Processing DeleteTablet for tablet d21d5e6d72c24e73b296c859773ff5df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:06:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37334
I20260924 03:06:36.594777 18982 tablet_replica.cc:330] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335: stopping tablet replica
I20260924 03:06:36.594841 18984 tablet_replica.cc:330] T d21d5e6d72c24e73b296c859773ff5df P 147a0711727045b39e5147066ec23d85: stopping tablet replica
I20260924 03:06:36.594910 18982 raft_consensus.cc:2229] T d21d5e6d72c24e73b296c859773ff5df P 1bab5dba75aa4175a3a3e9a22755b335 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:06:36.594921 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 18544
W20260924 03:06:36.599555 18468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40319 (error 108)
I20260924 03:06:36.599730 18984 raft_consensus.cc:2229] T d21d5e6d72c24e73b296c859773ff5df P 147a0711727045b39e5147066ec23d85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:06:36.599781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 18678
I20260924 03:06:36.599896 18984 raft_consensus.cc:2258] T d21d5e6d72c24e73b296c859773ff5df P 147a0711727045b39e5147066ec23d85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:06:36.600658 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:36.600057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40319 (local address 127.0.0.1:35484)
0924 03:06:36.600570 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:36.600615 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490}
I20260924 03:06:36.600853 18984 ts_tablet_manager.cc:1905] T d21d5e6d72c24e73b296c859773ff5df P 147a0711727045b39e5147066ec23d85: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 03:06:36.601542 18468 catalog_manager.cc:4584] TS 1bab5dba75aa4175a3a3e9a22755b335 (127.0.98.1:40319): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d21d5e6d72c24e73b296c859773ff5df: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40319: connect: Connection refused (error 111)
I20260924 03:06:36.602826 18984 ts_tablet_manager.cc:1918] T d21d5e6d72c24e73b296c859773ff5df P 147a0711727045b39e5147066ec23d85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:06:36.602892 18984 log.cc:1196] T d21d5e6d72c24e73b296c859773ff5df P 147a0711727045b39e5147066ec23d85: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218948286999-392-0/minicluster-data/ts-2/wal/wals/d21d5e6d72c24e73b296c859773ff5df
I20260924 03:06:36.603180 18984 ts_tablet_manager.cc:1939] T d21d5e6d72c24e73b296c859773ff5df P 147a0711727045b39e5147066ec23d85: Deleting consensus metadata
I20260924 03:06:36.603708 18469 catalog_manager.cc:4841] TS 147a0711727045b39e5147066ec23d85 (127.0.98.3:35653): tablet d21d5e6d72c24e73b296c859773ff5df (table test-table [id=635d334681e64851a33cd35a27d47755]) successfully deleted
I20260924 03:06:36.605031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 18812
I20260924 03:06:36.605194 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:36.605002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44097 (local address 127.0.0.1:49098)
0924 03:06:36.605083 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:36.605130 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 03:06:36.610455 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 18450
2026-09-24T03:06:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2247 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T03:06:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:06:36Z Disabled control of system clock
I20260924 03:06:36.640326 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34995
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:06:36.723142 18989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:36.723385 18989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:36.723407 18989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:36.724969 18989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:06:36.725028 18989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:36.725050 18989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:36.725075 18989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:06:36.725121 18989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:06:36.727210 18989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34995
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:36.727522 18989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:36.727880 18989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:36.733222 18995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:36.733348 18994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:36.733256 18989 server_base.cc:1034] running on GCE node
W20260924 03:06:36.733251 18997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:36.733814 18989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:36.734122 18989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:36.735281 18989 hybrid_clock.cc:648] HybridClock initialized: now 1790219196735264 us; error 40 us; skew 500 ppm
I20260924 03:06:36.736699 18989 webserver.cc:460] Webserver started at http://127.0.98.62:33581/ using document root <none> and password file <none>
I20260924 03:06:36.736981 18989 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:36.737032 18989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:36.737125 18989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:36.738272 18989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "0b18635e8b07409a89a2266e8f23faf5"
format_stamp: "Formatted at 2026-09-24 03:06:36 on dist-test-slave-wgsb"
server_key: "80ac0c149901e723102557fc825a975e"
server_key_iv: "2c6256e369ea186b543ce28f549ec7de"
server_key_version: "encryptionkey@0"
I20260924 03:06:36.738634 18989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "0b18635e8b07409a89a2266e8f23faf5"
format_stamp: "Formatted at 2026-09-24 03:06:36 on dist-test-slave-wgsb"
server_key: "80ac0c149901e723102557fc825a975e"
server_key_iv: "2c6256e369ea186b543ce28f549ec7de"
server_key_version: "encryptionkey@0"
I20260924 03:06:36.740031 18989 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:06:36.740805 19003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:36.740967 18989 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:06:36.741071 18989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "0b18635e8b07409a89a2266e8f23faf5"
format_stamp: "Formatted at 2026-09-24 03:06:36 on dist-test-slave-wgsb"
server_key: "80ac0c149901e723102557fc825a975e"
server_key_iv: "2c6256e369ea186b543ce28f549ec7de"
server_key_version: "encryptionkey@0"
I20260924 03:06:36.741146 18989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:36.772683 18989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:36.773059 18989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:36.773228 18989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:36.777495 18989 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34995
I20260924 03:06:36.777536 19055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34995 every 8 connection(s)
I20260924 03:06:36.777891 18989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:06:36.778870 19056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:36.781276 19056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b18635e8b07409a89a2266e8f23faf5: Bootstrap starting.
I20260924 03:06:36.781845 19056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b18635e8b07409a89a2266e8f23faf5: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:36.782099 19056 log.cc:824] T 00000000000000000000000000000000 P 0b18635e8b07409a89a2266e8f23faf5: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:36.782819 19056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b18635e8b07409a89a2266e8f23faf5: No bootstrap required, opened a new log
I20260924 03:06:36.784490 19056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b18635e8b07409a89a2266e8f23faf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b18635e8b07409a89a2266e8f23faf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34995 } }
I20260924 03:06:36.784636 19056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b18635e8b07409a89a2266e8f23faf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:36.784658 19056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b18635e8b07409a89a2266e8f23faf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b18635e8b07409a89a2266e8f23faf5, State: Initialized, Role: FOLLOWER
I20260924 03:06:36.784770 19056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b18635e8b07409a89a2266e8f23faf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b18635e8b07409a89a2266e8f23faf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34995 } }
I20260924 03:06:36.784852 19056 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b18635e8b07409a89a2266e8f23faf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:06:36.784888 19056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b18635e8b07409a89a2266e8f23faf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:06:36.784961 19056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b18635e8b07409a89a2266e8f23faf5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:36.785389 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 18989
I20260924 03:06:36.785475 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:06:36.785542 19056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b18635e8b07409a89a2266e8f23faf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b18635e8b07409a89a2266e8f23faf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34995 } }
I20260924 03:06:36.785667 19056 leader_election.cc:302] T 00000000000000000000000000000000 P 0b18635e8b07409a89a2266e8f23faf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b18635e8b07409a89a2266e8f23faf5; no voters:
I20260924 03:06:36.785805 392 external_mini_cluster.cc:1428] Setting key aa86263eb32bcd093a0f7dd6a870bd74
I20260924 03:06:36.785882 19056 leader_election.cc:288] T 00000000000000000000000000000000 P 0b18635e8b07409a89a2266e8f23faf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:06:36.785959 19059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b18635e8b07409a89a2266e8f23faf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:06:36.786164 19056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b18635e8b07409a89a2266e8f23faf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:06:36.786233 19059 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b18635e8b07409a89a2266e8f23faf5 [term 1 LEADER]: Becoming Leader. State: Replica: 0b18635e8b07409a89a2266e8f23faf5, State: Running, Role: LEADER
I20260924 03:06:36.786337 19059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b18635e8b07409a89a2266e8f23faf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b18635e8b07409a89a2266e8f23faf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34995 } }
I20260924 03:06:36.786675 19059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b18635e8b07409a89a2266e8f23faf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b18635e8b07409a89a2266e8f23faf5. Latest consensus state: current_term: 1 leader_uuid: "0b18635e8b07409a89a2266e8f23faf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b18635e8b07409a89a2266e8f23faf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34995 } } }
I20260924 03:06:36.786777 19059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b18635e8b07409a89a2266e8f23faf5 [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:36.786789 19060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b18635e8b07409a89a2266e8f23faf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b18635e8b07409a89a2266e8f23faf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b18635e8b07409a89a2266e8f23faf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34995 } } }
I20260924 03:06:36.786854 19060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b18635e8b07409a89a2266e8f23faf5 [sys.catalog]: This master's current role is: LEADER
W20260924 03:06:36.788265 19076 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b18635e8b07409a89a2266e8f23faf5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:06:36.788331 19076 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:06:36.788388 19069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:06:36.788573 19069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:06:36.791828 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:36.786383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32982 (local address 127.0.98.62:34995)
0924 03:06:36.786953 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:36.786962 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:36.787006 (+ 44us) server_negotiation.cc:407] Connection header received
0924 03:06:36.787105 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:36.787110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:36.787165 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:36.787318 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:36.787631 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.788992 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.789801 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.790223 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.790365 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:36.791107 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:36.791111 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:36.791112 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:36.791305 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:36.791338 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:36.791349 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:36.791425 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:36.791502 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 03:06:36.791584 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":68,"threads_started":1}
I20260924 03:06:36.792816 19069 catalog_manager.cc:1347] Generated new cluster ID: c7d714da4e9347a4a6abbca45af0d6a6
I20260924 03:06:36.792862 19069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:06:36.808079 19069 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:06:36.808632 19069 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:06:36.812958 19069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b18635e8b07409a89a2266e8f23faf5: Generated new TSK 0
I20260924 03:06:36.813268 19069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:06:36.817375 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34995
--builtin_ntp_servers=127.0.98.20:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:06:36.913998 19080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:36.914209 19080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:36.914232 19080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:36.915434 19080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:36.915479 19080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:36.915506 19080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:06:36.916947 19080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:36.917205 19080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:36.917534 19080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:36.923488 19086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:36.923504 19085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:36.923614 19088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:36.923857 19080 server_base.cc:1034] running on GCE node
I20260924 03:06:36.924080 19080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:36.924322 19080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:36.925489 19080 hybrid_clock.cc:648] HybridClock initialized: now 1790219196925458 us; error 41 us; skew 500 ppm
I20260924 03:06:36.926930 19080 webserver.cc:460] Webserver started at http://127.0.98.1:41583/ using document root <none> and password file <none>
I20260924 03:06:36.927218 19080 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:36.927279 19080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:36.927402 19080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:36.928884 19080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-0/data/instance:
uuid: "d91670ac1a684d288fbbd73e5b11920f"
format_stamp: "Formatted at 2026-09-24 03:06:36 on dist-test-slave-wgsb"
server_key: "32c8f5269ef11a0090e8a9bf82031630"
server_key_iv: "770339287595120a5c79b66a23a59296"
server_key_version: "encryptionkey@0"
I20260924 03:06:36.929280 19080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d91670ac1a684d288fbbd73e5b11920f"
format_stamp: "Formatted at 2026-09-24 03:06:36 on dist-test-slave-wgsb"
server_key: "32c8f5269ef11a0090e8a9bf82031630"
server_key_iv: "770339287595120a5c79b66a23a59296"
server_key_version: "encryptionkey@0"
I20260924 03:06:36.930888 19080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:06:36.931877 19094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:36.932111 19080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:06:36.932204 19080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "d91670ac1a684d288fbbd73e5b11920f"
format_stamp: "Formatted at 2026-09-24 03:06:36 on dist-test-slave-wgsb"
server_key: "32c8f5269ef11a0090e8a9bf82031630"
server_key_iv: "770339287595120a5c79b66a23a59296"
server_key_version: "encryptionkey@0"
I20260924 03:06:36.932273 19080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:36.953619 19080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:36.953908 19080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:36.954015 19080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:36.954212 19080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:06:36.954592 19080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:06:36.954649 19080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:36.954685 19080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:06:36.954733 19080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:36.960798 19080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35013
I20260924 03:06:36.960892 19207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35013 every 8 connection(s)
I20260924 03:06:36.961262 19080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:06:36.963474 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 19080
I20260924 03:06:36.963625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:06:36.963845 392 external_mini_cluster.cc:1428] Setting key 18e2df0cb4db302abac28395a8293c1a
I20260924 03:06:36.964912 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34995
--builtin_ntp_servers=127.0.98.20:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:06:36.967968 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:36.962056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34075 (local address 127.0.98.62:34995)
0924 03:06:36.962182 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:36.962184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:36.962353 (+ 169us) server_negotiation.cc:407] Connection header received
0924 03:06:36.962933 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:36.962935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:36.962948 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:36.962994 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:36.964023 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.964679 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.966251 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.966593 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.966719 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:36.967704 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:36.967705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:36.967706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:36.967710 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:36.967724 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:36.967728 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:36.967775 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:36.967879 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 03:06:36.967907 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:06:36.968256 19210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:36.962026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34995 (local address 127.0.98.1:34075)
0924 03:06:36.962213 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:36.962255 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:36.962825 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:36.963059 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:36.963063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:36.963072 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:06:36.963858 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:36.963865 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.964772 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.964773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:36.966116 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:36.966120 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:36.966726 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:36.966727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:36.966841 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:36.967474 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:06:36.967475 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:06:36.967515 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:06:36.967544 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:06:36.967794 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:06:36.967801 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:06:36.967804 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:06:36.967875 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 03:06:36.967994 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
I20260924 03:06:36.968894 19208 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34995
I20260924 03:06:36.969002 19208 heartbeater.cc:464] Registering TS with master...
I20260924 03:06:36.969225 19208 heartbeater.cc:511] Master 127.0.98.62:34995 requested a full tablet report, sending...
I20260924 03:06:36.969834 19020 ts_manager.cc:194] Registered new tserver with Master: d91670ac1a684d288fbbd73e5b11920f (127.0.98.1:35013)
I20260924 03:06:36.970908 19020 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:34075
W20260924 03:06:37.049434 19211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:37.049642 19211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:37.049664 19211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:37.050752 19211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:37.050788 19211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:37.050813 19211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:06:37.052171 19211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:37.052371 19211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:37.052618 19211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:37.057962 19219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:37.057964 19216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:37.057988 19217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:37.058461 19211 server_base.cc:1034] running on GCE node
I20260924 03:06:37.058696 19211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:37.058940 19211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:37.060144 19211 hybrid_clock.cc:648] HybridClock initialized: now 1790219197060091 us; error 61 us; skew 500 ppm
I20260924 03:06:37.061488 19211 webserver.cc:460] Webserver started at http://127.0.98.2:45869/ using document root <none> and password file <none>
I20260924 03:06:37.061769 19211 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:37.061846 19211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:37.061993 19211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:37.063063 19211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-1/data/instance:
uuid: "4cb479ead8c04e1b9bdb1621f35662e9"
format_stamp: "Formatted at 2026-09-24 03:06:37 on dist-test-slave-wgsb"
server_key: "aab930820022736cbaa5e8e6207e511f"
server_key_iv: "ca9ff1913eb2e59ba186dc4d7a340b5c"
server_key_version: "encryptionkey@0"
I20260924 03:06:37.063437 19211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4cb479ead8c04e1b9bdb1621f35662e9"
format_stamp: "Formatted at 2026-09-24 03:06:37 on dist-test-slave-wgsb"
server_key: "aab930820022736cbaa5e8e6207e511f"
server_key_iv: "ca9ff1913eb2e59ba186dc4d7a340b5c"
server_key_version: "encryptionkey@0"
I20260924 03:06:37.064874 19211 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:06:37.065709 19225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:37.065946 19211 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:06:37.066047 19211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "4cb479ead8c04e1b9bdb1621f35662e9"
format_stamp: "Formatted at 2026-09-24 03:06:37 on dist-test-slave-wgsb"
server_key: "aab930820022736cbaa5e8e6207e511f"
server_key_iv: "ca9ff1913eb2e59ba186dc4d7a340b5c"
server_key_version: "encryptionkey@0"
I20260924 03:06:37.066130 19211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:37.090250 19211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:37.090607 19211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:37.090771 19211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:37.091022 19211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:06:37.091358 19211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:06:37.091411 19211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:37.091447 19211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:06:37.091493 19211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:37.097373 19211 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38693
I20260924 03:06:37.097442 19338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38693 every 8 connection(s)
I20260924 03:06:37.097757 19211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:06:37.101241 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 19211
I20260924 03:06:37.101330 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:06:37.101532 392 external_mini_cluster.cc:1428] Setting key 80931aa82a085946908fc2cc0a547b35
I20260924 03:06:37.102670 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34995
--builtin_ntp_servers=127.0.98.20:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:06:37.103824 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:37.098583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56373 (local address 127.0.98.62:34995)
0924 03:06:37.098680 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:37.098683 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:37.098953 (+ 270us) server_negotiation.cc:407] Connection header received
0924 03:06:37.099519 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:37.099520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:37.099532 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:37.099622 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:37.100398 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.100922 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.102263 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.102464 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.102545 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:37.103510 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:37.103512 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:37.103513 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:37.103518 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:37.103536 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:37.103545 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:37.103635 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:37.103707 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 03:06:37.103743 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 03:06:37.104058 19341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:37.098582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34995 (local address 127.0.98.2:56373)
0924 03:06:37.098817 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:37.098852 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:37.099378 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:37.099738 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:37.099742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:37.099750 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:06:37.100267 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:37.100272 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.101043 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.101044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:37.102144 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:37.102147 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.102553 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.102554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:37.102694 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:37.103348 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:06:37.103348 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:06:37.103378 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:06:37.103402 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:06:37.103624 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:06:37.103630 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:06:37.103634 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:06:37.103711 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 03:06:37.103813 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":109,"threads_started":1}
I20260924 03:06:37.104534 19339 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34995
I20260924 03:06:37.104642 19339 heartbeater.cc:464] Registering TS with master...
I20260924 03:06:37.104827 19339 heartbeater.cc:511] Master 127.0.98.62:34995 requested a full tablet report, sending...
I20260924 03:06:37.105235 19020 ts_manager.cc:194] Registered new tserver with Master: 4cb479ead8c04e1b9bdb1621f35662e9 (127.0.98.2:38693)
I20260924 03:06:37.105614 19020 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:56373
W20260924 03:06:37.184810 19342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:37.185031 19342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:37.185055 19342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:37.186468 19342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:37.186507 19342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:37.186533 19342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:06:37.188053 19342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:37.188310 19342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:37.188621 19342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:37.193629 19348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:37.193626 19347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:37.193640 19350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:37.193913 19342 server_base.cc:1034] running on GCE node
I20260924 03:06:37.194195 19342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:37.194651 19342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:37.195925 19342 hybrid_clock.cc:648] HybridClock initialized: now 1790219197195885 us; error 50 us; skew 500 ppm
I20260924 03:06:37.197328 19342 webserver.cc:460] Webserver started at http://127.0.98.3:40153/ using document root <none> and password file <none>
I20260924 03:06:37.197577 19342 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:37.197677 19342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:37.197824 19342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:37.198925 19342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-2/data/instance:
uuid: "e9e6cbe4d707415299cfcf687d5bfbc7"
format_stamp: "Formatted at 2026-09-24 03:06:37 on dist-test-slave-wgsb"
server_key: "0f29bd5f4d2810bab837cdc01bd8dd03"
server_key_iv: "fc3d07f15119ec2a9c0c951c62c60946"
server_key_version: "encryptionkey@0"
I20260924 03:06:37.199302 19342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e9e6cbe4d707415299cfcf687d5bfbc7"
format_stamp: "Formatted at 2026-09-24 03:06:37 on dist-test-slave-wgsb"
server_key: "0f29bd5f4d2810bab837cdc01bd8dd03"
server_key_iv: "fc3d07f15119ec2a9c0c951c62c60946"
server_key_version: "encryptionkey@0"
I20260924 03:06:37.200695 19342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:06:37.201439 19356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:37.201608 19342 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:06:37.201709 19342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "e9e6cbe4d707415299cfcf687d5bfbc7"
format_stamp: "Formatted at 2026-09-24 03:06:37 on dist-test-slave-wgsb"
server_key: "0f29bd5f4d2810bab837cdc01bd8dd03"
server_key_iv: "fc3d07f15119ec2a9c0c951c62c60946"
server_key_version: "encryptionkey@0"
I20260924 03:06:37.201794 19342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:37.215054 19342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:37.215389 19342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:37.215552 19342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:37.215827 19342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:06:37.216181 19342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:06:37.216240 19342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:37.216280 19342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:06:37.216310 19342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:37.222246 19342 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41993
I20260924 03:06:37.222304 19469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41993 every 8 connection(s)
I20260924 03:06:37.222675 19342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:06:37.228071 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:37.223456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57951 (local address 127.0.98.62:34995)
0924 03:06:37.223530 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:37.223533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:37.223851 (+ 318us) server_negotiation.cc:407] Connection header received
0924 03:06:37.224376 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:37.224377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:37.224389 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:37.224459 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:37.225125 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.225634 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.226675 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.226875 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.226947 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:37.227820 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:37.227821 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:37.227821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:37.227825 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:37.227840 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:37.227844 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:37.227901 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:37.227979 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 03:06:37.228007 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 03:06:37.228291 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:37.223435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34995 (local address 127.0.98.3:57951)
0924 03:06:37.223724 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:37.223758 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:37.224234 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:37.224534 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:37.224538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:37.224545 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:06:37.225043 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:37.225047 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.225735 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.225746 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:37.226581 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:37.226583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.226980 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.226981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:37.227100 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:37.227660 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:06:37.227661 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:06:37.227698 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:06:37.227721 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:06:37.227908 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:06:37.227913 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:06:37.227915 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:06:37.227966 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 03:06:37.228045 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":100,"threads_started":1}
I20260924 03:06:37.228591 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 19342
I20260924 03:06:37.228744 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:06:37.228942 392 external_mini_cluster.cc:1428] Setting key 2503977567023a90921de7ea31f2f729
I20260924 03:06:37.228971 19470 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34995
I20260924 03:06:37.229249 19470 heartbeater.cc:464] Registering TS with master...
I20260924 03:06:37.229434 19470 heartbeater.cc:511] Master 127.0.98.62:34995 requested a full tablet report, sending...
I20260924 03:06:37.229818 19020 ts_manager.cc:194] Registered new tserver with Master: e9e6cbe4d707415299cfcf687d5bfbc7 (127.0.98.3:41993)
I20260924 03:06:37.230199 19020 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57951
I20260924 03:06:37.241293 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:06:37.246587 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:37.242963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32990 (local address 127.0.98.62:34995)
0924 03:06:37.243032 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:37.243034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:37.243042 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:06:37.243065 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:37.243066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:37.243076 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:37.243131 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:37.243909 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.244501 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.245363 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.245549 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.245607 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:37.246318 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:37.246319 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:37.246319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:37.246323 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:37.246338 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:37.246342 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:37.246420 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:37.246511 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 03:06:37.246523 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 03:06:37.248492 19020 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:06:37.250154 19020 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:06:37.257393 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:37.253155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58464 (local address 127.0.98.2:38693)
0924 03:06:37.253314 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:37.253318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:37.253589 (+ 271us) server_negotiation.cc:407] Connection header received
0924 03:06:37.253807 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:37.253809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:37.253827 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:37.253897 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:37.254139 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.254959 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.256433 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.256971 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.257064 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:37.257176 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:37.257255 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 03:06:37.257296 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260924 03:06:37.257390 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:37.253124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38693 (local address 127.0.0.1:58464)
0924 03:06:37.253513 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:37.253523 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:37.253576 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:37.253976 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:37.253977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:37.253978 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:37.254048 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:37.254050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.255081 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.255082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:37.256333 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:37.256335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.257062 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.257063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:37.257153 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:37.257168 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 03:06:37.257243 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 03:06:37.257297 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":25,"thread_start_us":133,"threads_started":1}
I20260924 03:06:37.257889 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:37.252799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41993 (local address 127.0.0.1:35790)
0924 03:06:37.253309 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:37.253358 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:37.253492 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:37.253848 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:37.253852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:37.253857 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:37.253975 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:37.253978 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.254969 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.254971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:37.256426 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:37.256429 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.257575 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.257576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:37.257665 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:37.257670 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:06:37.257756 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 03:06:37.257809 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"mutex_wait_us":28,"thread_start_us":77,"threads_started":1}
I20260924 03:06:37.257967 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:37.252992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35790 (local address 127.0.98.3:41993)
0924 03:06:37.253220 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:37.253225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:37.253441 (+ 216us) server_negotiation.cc:407] Connection header received
0924 03:06:37.253652 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:37.253656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:37.253677 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:37.253737 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:37.254071 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.254865 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.256497 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.257327 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.257433 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:37.257530 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:37.257801 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 03:06:37.257866 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":96,"threads_started":1}
I20260924 03:06:37.258373 19273 tablet_service.cc:1467] Processing CreateTablet for tablet f6f68426475747649f5fd61ae21473d2 (DEFAULT_TABLE table=test-table [id=0cd9c1641881487da7136fe03eead05a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:06:37.258776 19273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6f68426475747649f5fd61ae21473d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:37.258664 19404 tablet_service.cc:1467] Processing CreateTablet for tablet f6f68426475747649f5fd61ae21473d2 (DEFAULT_TABLE table=test-table [id=0cd9c1641881487da7136fe03eead05a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:06:37.259109 19404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6f68426475747649f5fd61ae21473d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:37.260931 19479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:37.252799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35013 (local address 127.0.0.1:46088)
0924 03:06:37.253374 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:37.253386 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:37.253650 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:37.253848 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:37.253852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:37.253854 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:37.253975 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:37.253977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.256206 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.256207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:37.257416 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:37.257419 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.260521 (+ 3102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.260523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:37.260639 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:37.260645 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 03:06:37.260724 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 03:06:37.260821 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":45,"spinlock_wait_cycles":384,"thread_start_us":73,"threads_started":1}
I20260924 03:06:37.260926 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:37.252898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46088 (local address 127.0.98.1:35013)
0924 03:06:37.253223 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:37.253230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:37.253460 (+ 230us) server_negotiation.cc:407] Connection header received
0924 03:06:37.253737 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:37.253740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:37.253757 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:37.253837 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:37.254991 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.256120 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.257740 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.260378 (+ 2638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.260503 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:37.260595 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:37.260778 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 03:06:37.260824 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":133,"threads_started":1}
I20260924 03:06:37.261756 19142 tablet_service.cc:1467] Processing CreateTablet for tablet f6f68426475747649f5fd61ae21473d2 (DEFAULT_TABLE table=test-table [id=0cd9c1641881487da7136fe03eead05a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:06:37.262102 19142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6f68426475747649f5fd61ae21473d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:37.262693 19484 tablet_bootstrap.cc:492] T f6f68426475747649f5fd61ae21473d2 P e9e6cbe4d707415299cfcf687d5bfbc7: Bootstrap starting.
I20260924 03:06:37.263001 19485 tablet_bootstrap.cc:492] T f6f68426475747649f5fd61ae21473d2 P 4cb479ead8c04e1b9bdb1621f35662e9: Bootstrap starting.
I20260924 03:06:37.263422 19484 tablet_bootstrap.cc:654] T f6f68426475747649f5fd61ae21473d2 P e9e6cbe4d707415299cfcf687d5bfbc7: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:37.263629 19485 tablet_bootstrap.cc:654] T f6f68426475747649f5fd61ae21473d2 P 4cb479ead8c04e1b9bdb1621f35662e9: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:37.263795 19484 log.cc:824] T f6f68426475747649f5fd61ae21473d2 P e9e6cbe4d707415299cfcf687d5bfbc7: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:37.263969 19485 log.cc:824] T f6f68426475747649f5fd61ae21473d2 P 4cb479ead8c04e1b9bdb1621f35662e9: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:37.264665 19484 tablet_bootstrap.cc:492] T f6f68426475747649f5fd61ae21473d2 P e9e6cbe4d707415299cfcf687d5bfbc7: No bootstrap required, opened a new log
I20260924 03:06:37.264757 19484 ts_tablet_manager.cc:1397] T f6f68426475747649f5fd61ae21473d2 P e9e6cbe4d707415299cfcf687d5bfbc7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 03:06:37.264770 19485 tablet_bootstrap.cc:492] T f6f68426475747649f5fd61ae21473d2 P 4cb479ead8c04e1b9bdb1621f35662e9: No bootstrap required, opened a new log
I20260924 03:06:37.264864 19485 ts_tablet_manager.cc:1397] T f6f68426475747649f5fd61ae21473d2 P 4cb479ead8c04e1b9bdb1621f35662e9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:06:37.265404 19488 tablet_bootstrap.cc:492] T f6f68426475747649f5fd61ae21473d2 P d91670ac1a684d288fbbd73e5b11920f: Bootstrap starting.
I20260924 03:06:37.266044 19488 tablet_bootstrap.cc:654] T f6f68426475747649f5fd61ae21473d2 P d91670ac1a684d288fbbd73e5b11920f: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:37.266353 19488 log.cc:824] T f6f68426475747649f5fd61ae21473d2 P d91670ac1a684d288fbbd73e5b11920f: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:37.267060 19488 tablet_bootstrap.cc:492] T f6f68426475747649f5fd61ae21473d2 P d91670ac1a684d288fbbd73e5b11920f: No bootstrap required, opened a new log
I20260924 03:06:37.267148 19488 ts_tablet_manager.cc:1397] T f6f68426475747649f5fd61ae21473d2 P d91670ac1a684d288fbbd73e5b11920f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:06:37.267647 19485 raft_consensus.cc:348] T f6f68426475747649f5fd61ae21473d2 P 4cb479ead8c04e1b9bdb1621f35662e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e6cbe4d707415299cfcf687d5bfbc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41993 } } peers { permanent_uuid: "d91670ac1a684d288fbbd73e5b11920f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35013 } } peers { permanent_uuid: "4cb479ead8c04e1b9bdb1621f35662e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38693 } }
I20260924 03:06:37.267652 19484 raft_consensus.cc:348] T f6f68426475747649f5fd61ae21473d2 P e9e6cbe4d707415299cfcf687d5bfbc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e6cbe4d707415299cfcf687d5bfbc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41993 } } peers { permanent_uuid: "d91670ac1a684d288fbbd73e5b11920f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35013 } } peers { permanent_uuid: "4cb479ead8c04e1b9bdb1621f35662e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38693 } }
I20260924 03:06:37.267833 19485 raft_consensus.cc:374] T f6f68426475747649f5fd61ae21473d2 P 4cb479ead8c04e1b9bdb1621f35662e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:37.267833 19484 raft_consensus.cc:374] T f6f68426475747649f5fd61ae21473d2 P e9e6cbe4d707415299cfcf687d5bfbc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:37.267871 19484 raft_consensus.cc:729] T f6f68426475747649f5fd61ae21473d2 P e9e6cbe4d707415299cfcf687d5bfbc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9e6cbe4d707415299cfcf687d5bfbc7, State: Initialized, Role: FOLLOWER
I20260924 03:06:37.267871 19485 raft_consensus.cc:729] T f6f68426475747649f5fd61ae21473d2 P 4cb479ead8c04e1b9bdb1621f35662e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cb479ead8c04e1b9bdb1621f35662e9, State: Initialized, Role: FOLLOWER
I20260924 03:06:37.268002 19485 consensus_queue.cc:260] T f6f68426475747649f5fd61ae21473d2 P 4cb479ead8c04e1b9bdb1621f35662e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e6cbe4d707415299cfcf687d5bfbc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41993 } } peers { permanent_uuid: "d91670ac1a684d288fbbd73e5b11920f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35013 } } peers { permanent_uuid: "4cb479ead8c04e1b9bdb1621f35662e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38693 } }
I20260924 03:06:37.268014 19484 consensus_queue.cc:260] T f6f68426475747649f5fd61ae21473d2 P e9e6cbe4d707415299cfcf687d5bfbc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e6cbe4d707415299cfcf687d5bfbc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41993 } } peers { permanent_uuid: "d91670ac1a684d288fbbd73e5b11920f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35013 } } peers { permanent_uuid: "4cb479ead8c04e1b9bdb1621f35662e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38693 } }
I20260924 03:06:37.268281 19484 ts_tablet_manager.cc:1428] T f6f68426475747649f5fd61ae21473d2 P e9e6cbe4d707415299cfcf687d5bfbc7: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 03:06:37.268281 19485 ts_tablet_manager.cc:1428] T f6f68426475747649f5fd61ae21473d2 P 4cb479ead8c04e1b9bdb1621f35662e9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:06:37.268318 19470 heartbeater.cc:503] Master 127.0.98.62:34995 was elected leader, sending a full tablet report...
I20260924 03:06:37.268404 19339 heartbeater.cc:503] Master 127.0.98.62:34995 was elected leader, sending a full tablet report...
I20260924 03:06:37.270299 19488 raft_consensus.cc:348] T f6f68426475747649f5fd61ae21473d2 P d91670ac1a684d288fbbd73e5b11920f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e6cbe4d707415299cfcf687d5bfbc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41993 } } peers { permanent_uuid: "d91670ac1a684d288fbbd73e5b11920f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35013 } } peers { permanent_uuid: "4cb479ead8c04e1b9bdb1621f35662e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38693 } }
I20260924 03:06:37.270462 19488 raft_consensus.cc:374] T f6f68426475747649f5fd61ae21473d2 P d91670ac1a684d288fbbd73e5b11920f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:37.270509 19488 raft_consensus.cc:729] T f6f68426475747649f5fd61ae21473d2 P d91670ac1a684d288fbbd73e5b11920f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d91670ac1a684d288fbbd73e5b11920f, State: Initialized, Role: FOLLOWER
I20260924 03:06:37.270625 19488 consensus_queue.cc:260] T f6f68426475747649f5fd61ae21473d2 P d91670ac1a684d288fbbd73e5b11920f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e6cbe4d707415299cfcf687d5bfbc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41993 } } peers { permanent_uuid: "d91670ac1a684d288fbbd73e5b11920f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35013 } } peers { permanent_uuid: "4cb479ead8c04e1b9bdb1621f35662e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38693 } }
I20260924 03:06:37.270900 19488 ts_tablet_manager.cc:1428] T f6f68426475747649f5fd61ae21473d2 P d91670ac1a684d288fbbd73e5b11920f: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260924 03:06:37.270988 19208 heartbeater.cc:503] Master 127.0.98.62:34995 was elected leader, sending a full tablet report...
W20260924 03:06:37.348376 19340 tablet.cc:2369] T f6f68426475747649f5fd61ae21473d2 P 4cb479ead8c04e1b9bdb1621f35662e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:06:37.461980 19209 tablet.cc:2369] T f6f68426475747649f5fd61ae21473d2 P d91670ac1a684d288fbbd73e5b11920f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:06:37.473346 19471 tablet.cc:2369] T f6f68426475747649f5fd61ae21473d2 P e9e6cbe4d707415299cfcf687d5bfbc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:06:37.579846 19492 raft_consensus.cc:482] T f6f68426475747649f5fd61ae21473d2 P d91670ac1a684d288fbbd73e5b11920f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:06:37.580000 19492 raft_consensus.cc:504] T f6f68426475747649f5fd61ae21473d2 P d91670ac1a684d288fbbd73e5b11920f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e6cbe4d707415299cfcf687d5bfbc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41993 } } peers { permanent_uuid: "d91670ac1a684d288fbbd73e5b11920f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35013 } } peers { permanent_uuid: "4cb479ead8c04e1b9bdb1621f35662e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38693 } }
I20260924 03:06:37.580399 19492 leader_election.cc:288] T f6f68426475747649f5fd61ae21473d2 P d91670ac1a684d288fbbd73e5b11920f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9e6cbe4d707415299cfcf687d5bfbc7 (127.0.98.3:41993), 4cb479ead8c04e1b9bdb1621f35662e9 (127.0.98.2:38693)
I20260924 03:06:37.585058 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:37.580652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41993 (local address 127.0.98.1:34851)
0924 03:06:37.580802 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:37.580814 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:37.580886 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:37.581525 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:37.581527 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:37.581529 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:37.581620 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:37.581623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.582980 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.582981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:37.584146 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:37.584149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.584753 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.584755 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:37.584848 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:37.584852 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:06:37.584913 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 03:06:37.584959 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":70,"threads_started":1}
I20260924 03:06:37.585048 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:37.580711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34851 (local address 127.0.98.3:41993)
0924 03:06:37.580760 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:37.580763 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:37.580892 (+ 129us) server_negotiation.cc:407] Connection header received
0924 03:06:37.580980 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:37.580981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:37.580990 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:37.581299 (+ 309us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:37.581736 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.582592 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.584249 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.584649 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.584748 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:37.584759 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:37.584932 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 03:06:37.584978 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 03:06:37.585409 19424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f68426475747649f5fd61ae21473d2" candidate_uuid: "d91670ac1a684d288fbbd73e5b11920f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9e6cbe4d707415299cfcf687d5bfbc7" is_pre_election: true
I20260924 03:06:37.585601 19424 raft_consensus.cc:2454] T f6f68426475747649f5fd61ae21473d2 P e9e6cbe4d707415299cfcf687d5bfbc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d91670ac1a684d288fbbd73e5b11920f in term 0.
I20260924 03:06:37.586151 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:37.580667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38693 (local address 127.0.98.1:40365)
0924 03:06:37.580806 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:37.580820 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:37.580898 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:37.581260 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:37.581261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:37.581264 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:37.581744 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:37.581750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.583444 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.583445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:37.584619 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:37.584626 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.585891 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.585892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:37.586014 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:37.586018 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:06:37.586057 (+ 39us) client_negotiation.cc:240] Negotiation successful
0924 03:06:37.586090 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260924 03:06:37.586304 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:37.580766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40365 (local address 127.0.98.2:38693)
0924 03:06:37.580813 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:37.580815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:37.580865 (+ 50us) server_negotiation.cc:407] Connection header received
0924 03:06:37.580973 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:37.580974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:37.580991 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:37.581037 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:37.582476 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.583076 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.584906 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.585717 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.585884 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:37.585901 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:37.586111 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 03:06:37.586211 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 03:06:37.586192 19097 leader_election.cc:302] T f6f68426475747649f5fd61ae21473d2 P d91670ac1a684d288fbbd73e5b11920f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d91670ac1a684d288fbbd73e5b11920f, e9e6cbe4d707415299cfcf687d5bfbc7; no voters:
I20260924 03:06:37.586702 19492 raft_consensus.cc:2781] T f6f68426475747649f5fd61ae21473d2 P d91670ac1a684d288fbbd73e5b11920f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:06:37.586820 19492 raft_consensus.cc:482] T f6f68426475747649f5fd61ae21473d2 P d91670ac1a684d288fbbd73e5b11920f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:06:37.586869 19492 raft_consensus.cc:3037] T f6f68426475747649f5fd61ae21473d2 P d91670ac1a684d288fbbd73e5b11920f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:37.586871 19293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f68426475747649f5fd61ae21473d2" candidate_uuid: "d91670ac1a684d288fbbd73e5b11920f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cb479ead8c04e1b9bdb1621f35662e9" is_pre_election: true
I20260924 03:06:37.587127 19293 raft_consensus.cc:2454] T f6f68426475747649f5fd61ae21473d2 P 4cb479ead8c04e1b9bdb1621f35662e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d91670ac1a684d288fbbd73e5b11920f in term 0.
I20260924 03:06:37.588171 19492 raft_consensus.cc:504] T f6f68426475747649f5fd61ae21473d2 P d91670ac1a684d288fbbd73e5b11920f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e6cbe4d707415299cfcf687d5bfbc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41993 } } peers { permanent_uuid: "d91670ac1a684d288fbbd73e5b11920f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35013 } } peers { permanent_uuid: "4cb479ead8c04e1b9bdb1621f35662e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38693 } }
I20260924 03:06:37.588526 19492 leader_election.cc:288] T f6f68426475747649f5fd61ae21473d2 P d91670ac1a684d288fbbd73e5b11920f [CANDIDATE]: Term 1 election: Requested vote from peers e9e6cbe4d707415299cfcf687d5bfbc7 (127.0.98.3:41993), 4cb479ead8c04e1b9bdb1621f35662e9 (127.0.98.2:38693)
I20260924 03:06:37.588706 19424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f68426475747649f5fd61ae21473d2" candidate_uuid: "d91670ac1a684d288fbbd73e5b11920f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9e6cbe4d707415299cfcf687d5bfbc7"
I20260924 03:06:37.588706 19293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f68426475747649f5fd61ae21473d2" candidate_uuid: "d91670ac1a684d288fbbd73e5b11920f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cb479ead8c04e1b9bdb1621f35662e9"
I20260924 03:06:37.588829 19424 raft_consensus.cc:3037] T f6f68426475747649f5fd61ae21473d2 P e9e6cbe4d707415299cfcf687d5bfbc7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:37.588829 19293 raft_consensus.cc:3037] T f6f68426475747649f5fd61ae21473d2 P 4cb479ead8c04e1b9bdb1621f35662e9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:37.590085 19293 raft_consensus.cc:2454] T f6f68426475747649f5fd61ae21473d2 P 4cb479ead8c04e1b9bdb1621f35662e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d91670ac1a684d288fbbd73e5b11920f in term 1.
I20260924 03:06:37.590085 19424 raft_consensus.cc:2454] T f6f68426475747649f5fd61ae21473d2 P e9e6cbe4d707415299cfcf687d5bfbc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d91670ac1a684d288fbbd73e5b11920f in term 1.
I20260924 03:06:37.590343 19097 leader_election.cc:302] T f6f68426475747649f5fd61ae21473d2 P d91670ac1a684d288fbbd73e5b11920f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d91670ac1a684d288fbbd73e5b11920f, e9e6cbe4d707415299cfcf687d5bfbc7; no voters:
I20260924 03:06:37.590512 19492 raft_consensus.cc:2781] T f6f68426475747649f5fd61ae21473d2 P d91670ac1a684d288fbbd73e5b11920f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:06:37.590709 19492 raft_consensus.cc:686] T f6f68426475747649f5fd61ae21473d2 P d91670ac1a684d288fbbd73e5b11920f [term 1 LEADER]: Becoming Leader. State: Replica: d91670ac1a684d288fbbd73e5b11920f, State: Running, Role: LEADER
I20260924 03:06:37.590831 19492 consensus_queue.cc:237] T f6f68426475747649f5fd61ae21473d2 P d91670ac1a684d288fbbd73e5b11920f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e6cbe4d707415299cfcf687d5bfbc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41993 } } peers { permanent_uuid: "d91670ac1a684d288fbbd73e5b11920f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35013 } } peers { permanent_uuid: "4cb479ead8c04e1b9bdb1621f35662e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38693 } }
I20260924 03:06:37.591840 19020 catalog_manager.cc:5495] T f6f68426475747649f5fd61ae21473d2 P d91670ac1a684d288fbbd73e5b11920f reported cstate change: term changed from 0 to 1, leader changed from <none> to d91670ac1a684d288fbbd73e5b11920f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d91670ac1a684d288fbbd73e5b11920f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e6cbe4d707415299cfcf687d5bfbc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d91670ac1a684d288fbbd73e5b11920f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cb479ead8c04e1b9bdb1621f35662e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38693 } health_report { overall_health: UNKNOWN } } }
I20260924 03:06:37.617722 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:37.614732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46092 (local address 127.0.98.1:35013)
0924 03:06:37.614805 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:37.614806 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:37.614836 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:06:37.614874 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:37.614875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:37.614896 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:37.614957 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:06:37.615238 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.615848 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.616813 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:37.616996 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:37.617120 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:37.617327 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:06:37.617417 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:06:37.617495 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:37.617593 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 03:06:37.617607 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 03:06:37.620100 19424 raft_consensus.cc:1261] T f6f68426475747649f5fd61ae21473d2 P e9e6cbe4d707415299cfcf687d5bfbc7 [term 1 FOLLOWER]: Refusing update from remote peer d91670ac1a684d288fbbd73e5b11920f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:06:37.620152 19293 raft_consensus.cc:1261] T f6f68426475747649f5fd61ae21473d2 P 4cb479ead8c04e1b9bdb1621f35662e9 [term 1 FOLLOWER]: Refusing update from remote peer d91670ac1a684d288fbbd73e5b11920f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:06:37.620550 19492 consensus_queue.cc:1035] T f6f68426475747649f5fd61ae21473d2 P d91670ac1a684d288fbbd73e5b11920f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cb479ead8c04e1b9bdb1621f35662e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:06:37.620735 19495 consensus_queue.cc:1035] T f6f68426475747649f5fd61ae21473d2 P d91670ac1a684d288fbbd73e5b11920f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9e6cbe4d707415299cfcf687d5bfbc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:06:37.624684 19500 mvcc.cc:205] Tried to move back new op lower bound from 7332737833447710720 to 7332737833332977664. Current Snapshot: MvccSnapshot[applied={T|T < 7332737833447710720}]
I20260924 03:06:37.626223 19019 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:06:37.626314 19019 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:06:37.627246 19019 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b18635e8b07409a89a2266e8f23faf5: Sending DeleteTablet for 3 replicas of tablet f6f68426475747649f5fd61ae21473d2
I20260924 03:06:37.628232 19404 tablet_service.cc:1514] Processing DeleteTablet for tablet f6f68426475747649f5fd61ae21473d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:06:37 UTC) from {username='slave'} at 127.0.0.1:35790
I20260924 03:06:37.628700 19508 tablet_replica.cc:330] T f6f68426475747649f5fd61ae21473d2 P e9e6cbe4d707415299cfcf687d5bfbc7: stopping tablet replica
I20260924 03:06:37.628549 19273 tablet_service.cc:1514] Processing DeleteTablet for tablet f6f68426475747649f5fd61ae21473d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:06:37 UTC) from {username='slave'} at 127.0.0.1:58464
I20260924 03:06:37.628814 19508 raft_consensus.cc:2229] T f6f68426475747649f5fd61ae21473d2 P e9e6cbe4d707415299cfcf687d5bfbc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:06:37.628886 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 19080
I20260924 03:06:37.628928 19508 raft_consensus.cc:2258] T f6f68426475747649f5fd61ae21473d2 P e9e6cbe4d707415299cfcf687d5bfbc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:06:37.630273 19508 ts_tablet_manager.cc:1905] T f6f68426475747649f5fd61ae21473d2 P e9e6cbe4d707415299cfcf687d5bfbc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:06:37.630653 19509 tablet_replica.cc:330] T f6f68426475747649f5fd61ae21473d2 P 4cb479ead8c04e1b9bdb1621f35662e9: stopping tablet replica
I20260924 03:06:37.630776 19509 raft_consensus.cc:2229] T f6f68426475747649f5fd61ae21473d2 P 4cb479ead8c04e1b9bdb1621f35662e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:06:37.630890 19509 raft_consensus.cc:2258] T f6f68426475747649f5fd61ae21473d2 P 4cb479ead8c04e1b9bdb1621f35662e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:06:37.631407 19509 ts_tablet_manager.cc:1905] T f6f68426475747649f5fd61ae21473d2 P 4cb479ead8c04e1b9bdb1621f35662e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:06:37.632795 19508 ts_tablet_manager.cc:1918] T f6f68426475747649f5fd61ae21473d2 P e9e6cbe4d707415299cfcf687d5bfbc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:06:37.632880 19508 log.cc:1196] T f6f68426475747649f5fd61ae21473d2 P e9e6cbe4d707415299cfcf687d5bfbc7: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-2/wal/wals/f6f68426475747649f5fd61ae21473d2
I20260924 03:06:37.633169 19508 ts_tablet_manager.cc:1939] T f6f68426475747649f5fd61ae21473d2 P e9e6cbe4d707415299cfcf687d5bfbc7: Deleting consensus metadata
I20260924 03:06:37.633380 19509 ts_tablet_manager.cc:1918] T f6f68426475747649f5fd61ae21473d2 P 4cb479ead8c04e1b9bdb1621f35662e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:06:37.633450 19509 log.cc:1196] T f6f68426475747649f5fd61ae21473d2 P 4cb479ead8c04e1b9bdb1621f35662e9: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218948286999-392-0/minicluster-data/ts-1/wal/wals/f6f68426475747649f5fd61ae21473d2
I20260924 03:06:37.633653 19006 catalog_manager.cc:4841] TS e9e6cbe4d707415299cfcf687d5bfbc7 (127.0.98.3:41993): tablet f6f68426475747649f5fd61ae21473d2 (table test-table [id=0cd9c1641881487da7136fe03eead05a]) successfully deleted
I20260924 03:06:37.633713 19509 ts_tablet_manager.cc:1939] T f6f68426475747649f5fd61ae21473d2 P 4cb479ead8c04e1b9bdb1621f35662e9: Deleting consensus metadata
I20260924 03:06:37.634218 19005 catalog_manager.cc:4841] TS 4cb479ead8c04e1b9bdb1621f35662e9 (127.0.98.2:38693): tablet f6f68426475747649f5fd61ae21473d2 (table test-table [id=0cd9c1641881487da7136fe03eead05a]) successfully deleted
W20260924 03:06:37.634886 19005 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35013 (error 108)
I20260924 03:06:37.635485 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 19211
I20260924 03:06:37.635538 19479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:37.635375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35013 (local address 127.0.0.1:46100)
0924 03:06:37.635439 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:37.635498 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 03:06:37.636554 19005 catalog_manager.cc:4584] TS d91670ac1a684d288fbbd73e5b11920f (127.0.98.1:35013): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f6f68426475747649f5fd61ae21473d2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35013: connect: Connection refused (error 111)
I20260924 03:06:37.641762 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 19342
I20260924 03:06:37.648497 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 18989
2026-09-24T03:06:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1036 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:06:37 dist-test-slave-wgsb krb5kdc[19513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:06:37 dist-test-slave-wgsb krb5kdc[19513](info): set up 2 sockets
Sep 24 03:06:37 dist-test-slave-wgsb krb5kdc[19513](info): commencing operation
krb5kdc: starting...
W20260924 03:06:38.715547 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:06:38 dist-test-slave-wgsb krb5kdc[19513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:06:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:06:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:06:38.839491 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35187
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:06:38.921556 19526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:38.921794 19526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:38.921818 19526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:38.922991 19526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:06:38.923032 19526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:38.923046 19526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:38.923059 19526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:06:38.923071 19526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:06:38.924618 19526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:38.924876 19526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:38.925109 19526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:38.930430 19531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:38.930449 19526 server_base.cc:1034] running on GCE node
W20260924 03:06:38.930466 19532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:38.930430 19534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:38.930994 19526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:38.931308 19526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:38.932492 19526 hybrid_clock.cc:648] HybridClock initialized: now 1790219198932478 us; error 54 us; skew 500 ppm
Sep 24 03:06:38 dist-test-slave-wgsb krb5kdc[19513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:38.935083 19526 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:06:38.935559 19526 webserver.cc:460] Webserver started at http://127.0.98.62:41813/ using document root <none> and password file <none>
I20260924 03:06:38.935842 19526 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:38.935911 19526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:38.936053 19526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:38.937126 19526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "9ded570286a8464e86c9c3136fbd0924"
format_stamp: "Formatted at 2026-09-24 03:06:38 on dist-test-slave-wgsb"
server_key: "882173aa9c09414e32d430ed1c13bef7"
server_key_iv: "dedabe1ca4d09139ddedac78564ceba2"
server_key_version: "encryptionkey@0"
I20260924 03:06:38.937477 19526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "9ded570286a8464e86c9c3136fbd0924"
format_stamp: "Formatted at 2026-09-24 03:06:38 on dist-test-slave-wgsb"
server_key: "882173aa9c09414e32d430ed1c13bef7"
server_key_iv: "dedabe1ca4d09139ddedac78564ceba2"
server_key_version: "encryptionkey@0"
I20260924 03:06:38.938771 19526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:06:38.939497 19541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:38.939711 19526 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:06:38.939800 19526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "9ded570286a8464e86c9c3136fbd0924"
format_stamp: "Formatted at 2026-09-24 03:06:38 on dist-test-slave-wgsb"
server_key: "882173aa9c09414e32d430ed1c13bef7"
server_key_iv: "dedabe1ca4d09139ddedac78564ceba2"
server_key_version: "encryptionkey@0"
I20260924 03:06:38.939880 19526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:38.976507 19526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:38.978652 19526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:38.978821 19526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:38.983137 19526 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35187
I20260924 03:06:38.983188 19593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35187 every 8 connection(s)
I20260924 03:06:38.983566 19526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:06:38.984503 19594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:38.984786 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 19526
I20260924 03:06:38.984911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:06:38.985163 392 external_mini_cluster.cc:1428] Setting key a20b5980b6236b6418fe1ac7363994dd
I20260924 03:06:38.987674 19594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ded570286a8464e86c9c3136fbd0924: Bootstrap starting.
I20260924 03:06:38.988355 19594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ded570286a8464e86c9c3136fbd0924: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:38.988672 19594 log.cc:824] T 00000000000000000000000000000000 P 9ded570286a8464e86c9c3136fbd0924: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:38.989439 19594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ded570286a8464e86c9c3136fbd0924: No bootstrap required, opened a new log
I20260924 03:06:38.991971 19594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ded570286a8464e86c9c3136fbd0924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ded570286a8464e86c9c3136fbd0924" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35187 } }
I20260924 03:06:38.992178 19594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ded570286a8464e86c9c3136fbd0924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:38.992213 19594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ded570286a8464e86c9c3136fbd0924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ded570286a8464e86c9c3136fbd0924, State: Initialized, Role: FOLLOWER
I20260924 03:06:38.992334 19594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ded570286a8464e86c9c3136fbd0924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ded570286a8464e86c9c3136fbd0924" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35187 } }
I20260924 03:06:38.992429 19594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ded570286a8464e86c9c3136fbd0924 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:06:38.992466 19594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ded570286a8464e86c9c3136fbd0924 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:06:38.992527 19594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ded570286a8464e86c9c3136fbd0924 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 03:06:38 dist-test-slave-wgsb krb5kdc[19513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:06:38.993276 19594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ded570286a8464e86c9c3136fbd0924 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ded570286a8464e86c9c3136fbd0924" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35187 } }
I20260924 03:06:38.993394 19594 leader_election.cc:302] T 00000000000000000000000000000000 P 9ded570286a8464e86c9c3136fbd0924 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ded570286a8464e86c9c3136fbd0924; no voters:
I20260924 03:06:38.993597 19594 leader_election.cc:288] T 00000000000000000000000000000000 P 9ded570286a8464e86c9c3136fbd0924 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:06:38.993683 19599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ded570286a8464e86c9c3136fbd0924 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:06:38.993837 19599 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ded570286a8464e86c9c3136fbd0924 [term 1 LEADER]: Becoming Leader. State: Replica: 9ded570286a8464e86c9c3136fbd0924, State: Running, Role: LEADER
I20260924 03:06:38.993943 19594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ded570286a8464e86c9c3136fbd0924 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:06:38.994007 19599 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ded570286a8464e86c9c3136fbd0924 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ded570286a8464e86c9c3136fbd0924" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35187 } }
I20260924 03:06:38.994460 19600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ded570286a8464e86c9c3136fbd0924 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ded570286a8464e86c9c3136fbd0924" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ded570286a8464e86c9c3136fbd0924" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35187 } } }
I20260924 03:06:38.994475 19601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ded570286a8464e86c9c3136fbd0924 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ded570286a8464e86c9c3136fbd0924. Latest consensus state: current_term: 1 leader_uuid: "9ded570286a8464e86c9c3136fbd0924" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ded570286a8464e86c9c3136fbd0924" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35187 } } }
I20260924 03:06:38.994616 19600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ded570286a8464e86c9c3136fbd0924 [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:38.995426 19608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:06:38.996531 19608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:06:38.994710 19601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ded570286a8464e86c9c3136fbd0924 [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:38.997646 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:38.985744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51256 (local address 127.0.98.62:35187)
0924 03:06:38.985966 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:38.985975 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:38.986013 (+ 38us) server_negotiation.cc:407] Connection header received
0924 03:06:38.986566 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:38.986575 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:38.986635 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:38.986782 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:38.987399 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:38.988800 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:38.989965 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:38.990400 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:38.990492 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:38.993692 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:38.993697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:38.993697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:06:38.993710 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:38.995937 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:38.996343 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:38.996347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:38.996348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:38.996455 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:38.996686 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:38.996687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:38.996687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:38.996981 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:38.997131 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:38.997284 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 03:06:38.997439 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":99,"threads_started":1}
I20260924 03:06:38.998127 19608 catalog_manager.cc:1347] Generated new cluster ID: 13ff6f596cbc4de69bf2a2176d3d6ae1
I20260924 03:06:38.998181 19608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:06:39.011425 19608 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:06:39.012043 19608 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:06:39.019578 19608 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ded570286a8464e86c9c3136fbd0924: Generated new TSK 0
I20260924 03:06:39.019868 19608 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:06:39.046797 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35187
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:06:39.129307 19620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:39.129536 19620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:39.129572 19620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:39.130699 19620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:39.130761 19620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:39.130807 19620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:06:39.132400 19620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:39.132668 19620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:39.132894 19620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:39.137955 19626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:39.137919 19628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:39.137913 19625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:39.138312 19620 server_base.cc:1034] running on GCE node
I20260924 03:06:39.138631 19620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:39.138856 19620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:39.140015 19620 hybrid_clock.cc:648] HybridClock initialized: now 1790219199139997 us; error 28 us; skew 500 ppm
Sep 24 03:06:39 dist-test-slave-wgsb krb5kdc[19513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:39.143028 19620 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:06:39.143554 19620 webserver.cc:460] Webserver started at http://127.0.98.1:36125/ using document root <none> and password file <none>
I20260924 03:06:39.143810 19620 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:39.143880 19620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:39.144026 19620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:39.145113 19620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-0/data/instance:
uuid: "39d3b539fdf5441f8bd5a43ba4dd53cd"
format_stamp: "Formatted at 2026-09-24 03:06:39 on dist-test-slave-wgsb"
server_key: "bf28b226ecd92d1c7810b05d20a51fff"
server_key_iv: "33eaa498acefae4c62d43db2e22ebd57"
server_key_version: "encryptionkey@0"
I20260924 03:06:39.145476 19620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-0/wal/instance:
uuid: "39d3b539fdf5441f8bd5a43ba4dd53cd"
format_stamp: "Formatted at 2026-09-24 03:06:39 on dist-test-slave-wgsb"
server_key: "bf28b226ecd92d1c7810b05d20a51fff"
server_key_iv: "33eaa498acefae4c62d43db2e22ebd57"
server_key_version: "encryptionkey@0"
I20260924 03:06:39.146847 19620 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:06:39.147691 19635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:39.147874 19620 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:06:39.147971 19620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "39d3b539fdf5441f8bd5a43ba4dd53cd"
format_stamp: "Formatted at 2026-09-24 03:06:39 on dist-test-slave-wgsb"
server_key: "bf28b226ecd92d1c7810b05d20a51fff"
server_key_iv: "33eaa498acefae4c62d43db2e22ebd57"
server_key_version: "encryptionkey@0"
I20260924 03:06:39.148061 19620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:39.158391 19620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:39.161020 19620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:39.161139 19620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:39.161340 19620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:06:39.161676 19620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:06:39.161732 19620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:39.161777 19620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:06:39.161803 19620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:39.167428 19620 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38599
I20260924 03:06:39.167546 19748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38599 every 8 connection(s)
I20260924 03:06:39.167884 19620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:06:39.172174 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 19620
I20260924 03:06:39.172313 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:06:39.172530 392 external_mini_cluster.cc:1428] Setting key 9502980cc6f30736523a9a770a8f35d5
Sep 24 03:06:39 dist-test-slave-wgsb krb5kdc[19513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:06:39.179328 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:39.168698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52917 (local address 127.0.98.62:35187)
0924 03:06:39.168770 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:39.168772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:39.169006 (+ 234us) server_negotiation.cc:407] Connection header received
0924 03:06:39.169958 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:39.169959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:39.169969 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:39.170024 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:39.170659 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.171247 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.172539 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.172885 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.172965 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:39.175730 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:39.175732 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:39.175733 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:06:39.175740 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:39.177399 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:39.177882 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:39.177884 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:39.177885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:39.177949 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:39.178477 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:39.178479 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:39.178479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:39.178660 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:39.178745 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:39.179069 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 03:06:39.179185 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 03:06:39.179349 19751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:39.168687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35187 (local address 127.0.98.1:52917)
0924 03:06:39.168883 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:39.168920 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:39.169860 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:39.170098 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:39.170101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:39.170358 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:06:39.170543 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:39.170546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.171339 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.171340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:39.172457 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:39.172461 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.172982 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.172983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:39.173089 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:39.173722 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:06:39.173733 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:06:39.175575 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:06:39.177538 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:06:39.177547 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:06:39.177549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:06:39.177802 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:06:39.178044 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:06:39.178045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:06:39.178045 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:06:39.178390 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:06:39.178772 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:06:39.178778 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:06:39.178917 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 03:06:39.179026 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 03:06:39.179174 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":105,"threads_started":1}
I20260924 03:06:39.179977 19749 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35187
I20260924 03:06:39.180101 19749 heartbeater.cc:464] Registering TS with master...
I20260924 03:06:39.180290 19749 heartbeater.cc:511] Master 127.0.98.62:35187 requested a full tablet report, sending...
I20260924 03:06:39.180835 19558 ts_manager.cc:194] Registered new tserver with Master: 39d3b539fdf5441f8bd5a43ba4dd53cd (127.0.98.1:38599)
I20260924 03:06:39.181473 19558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52917
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:06:39.200332 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35187
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:06:39.282184 19754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:39.282467 19754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:39.282509 19754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:39.283720 19754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:39.283787 19754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:39.283831 19754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:06:39.285396 19754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:39.285676 19754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:39.285921 19754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:06:39.291390 19754 server_base.cc:1034] running on GCE node
W20260924 03:06:39.291323 19760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:39.291323 19762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:39.291361 19759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:39.291812 19754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:39.292119 19754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:39.293323 19754 hybrid_clock.cc:648] HybridClock initialized: now 1790219199293296 us; error 39 us; skew 500 ppm
Sep 24 03:06:39 dist-test-slave-wgsb krb5kdc[19513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:39.296605 19754 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:06:39.297113 19754 webserver.cc:460] Webserver started at http://127.0.98.2:39017/ using document root <none> and password file <none>
I20260924 03:06:39.297344 19754 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:39.297411 19754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:39.297561 19754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:39.298592 19754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-1/data/instance:
uuid: "ce0a1d07cc71410ab66b39c103392fdf"
format_stamp: "Formatted at 2026-09-24 03:06:39 on dist-test-slave-wgsb"
server_key: "2b5914d90deffe20fe9873d544e43288"
server_key_iv: "32c450bc68fe5a5c92c4a13ea875bb3e"
server_key_version: "encryptionkey@0"
I20260924 03:06:39.298949 19754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ce0a1d07cc71410ab66b39c103392fdf"
format_stamp: "Formatted at 2026-09-24 03:06:39 on dist-test-slave-wgsb"
server_key: "2b5914d90deffe20fe9873d544e43288"
server_key_iv: "32c450bc68fe5a5c92c4a13ea875bb3e"
server_key_version: "encryptionkey@0"
I20260924 03:06:39.300379 19754 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:06:39.301182 19769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:39.301383 19754 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 03:06:39.301486 19754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "ce0a1d07cc71410ab66b39c103392fdf"
format_stamp: "Formatted at 2026-09-24 03:06:39 on dist-test-slave-wgsb"
server_key: "2b5914d90deffe20fe9873d544e43288"
server_key_iv: "32c450bc68fe5a5c92c4a13ea875bb3e"
server_key_version: "encryptionkey@0"
I20260924 03:06:39.301559 19754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:39.314420 19754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:39.316898 19754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:39.317081 19754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:39.317381 19754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:06:39.317714 19754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:06:39.317770 19754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:39.317806 19754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:06:39.317853 19754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:39.324251 19754 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44183
I20260924 03:06:39.324342 19882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44183 every 8 connection(s)
I20260924 03:06:39.324816 19754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:06:39.325516 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 19754
I20260924 03:06:39.325634 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:06:39.325848 392 external_mini_cluster.cc:1428] Setting key 01733ef327c5d40ad4b259ff6ece18a2
Sep 24 03:06:39 dist-test-slave-wgsb krb5kdc[19513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 03:06:39.337214 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:39.325705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39919 (local address 127.0.98.62:35187)
0924 03:06:39.325774 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:39.325777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:39.325983 (+ 206us) server_negotiation.cc:407] Connection header received
0924 03:06:39.327396 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:39.327398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:39.327411 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:39.327472 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:39.328265 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.328945 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.330059 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.330408 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.330494 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:39.333699 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:39.333701 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:39.333701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:06:39.333711 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:39.335363 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:39.335925 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:39.335926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:39.335927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:39.335993 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:39.336478 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:39.336479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:39.336479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:39.336629 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:39.336710 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:39.337018 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 03:06:39.337115 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 03:06:39.337425 19885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:39.325632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35187 (local address 127.0.98.2:39919)
0924 03:06:39.325827 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:39.325882 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:39.327258 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:39.327552 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:39.327557 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:39.327926 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:06:39.328154 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:39.328158 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.329058 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.329059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:39.329950 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:39.329953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.330582 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.330584 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:39.330782 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:39.331846 (+ 1064us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:06:39.331857 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:06:39.333582 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:06:39.335486 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:06:39.335497 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:06:39.335499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:06:39.335827 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:06:39.336070 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:06:39.336071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:06:39.336071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:06:39.336214 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:06:39.336712 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:06:39.336720 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:06:39.336876 (+ 156us) client_negotiation.cc:769] Sending connection context
0924 03:06:39.336984 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 03:06:39.337174 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":113,"threads_started":1}
I20260924 03:06:39.338025 19883 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35187
I20260924 03:06:39.338186 19883 heartbeater.cc:464] Registering TS with master...
I20260924 03:06:39.338409 19883 heartbeater.cc:511] Master 127.0.98.62:35187 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 03:06:39.338840 19558 ts_manager.cc:194] Registered new tserver with Master: ce0a1d07cc71410ab66b39c103392fdf (127.0.98.2:44183)
I20260924 03:06:39.339341 19558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:06:39.356055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35187
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:06:39.454056 19888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:39.454320 19888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:39.454348 19888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:39.455825 19888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:39.455883 19888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:39.455915 19888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:06:39.457672 19888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:39.458077 19888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:39.458385 19888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:39.465368 19894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:39.465375 19893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:39.465382 19896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:39.465461 19888 server_base.cc:1034] running on GCE node
I20260924 03:06:39.466218 19888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:39.466566 19888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:39.467772 19888 hybrid_clock.cc:648] HybridClock initialized: now 1790219199467681 us; error 108 us; skew 500 ppm
Sep 24 03:06:39 dist-test-slave-wgsb krb5kdc[19513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:39.471791 19888 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:06:39.472412 19888 webserver.cc:460] Webserver started at http://127.0.98.3:34751/ using document root <none> and password file <none>
I20260924 03:06:39.472675 19888 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:39.472750 19888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:39.472908 19888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:39.474175 19888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-2/data/instance:
uuid: "01970f3378f54466a9b144e604b5fb2a"
format_stamp: "Formatted at 2026-09-24 03:06:39 on dist-test-slave-wgsb"
server_key: "155cd8de40e5fd01ee5004e80d47bba4"
server_key_iv: "8edd7f4020af99f6b29a19dba140dec3"
server_key_version: "encryptionkey@0"
I20260924 03:06:39.474586 19888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-2/wal/instance:
uuid: "01970f3378f54466a9b144e604b5fb2a"
format_stamp: "Formatted at 2026-09-24 03:06:39 on dist-test-slave-wgsb"
server_key: "155cd8de40e5fd01ee5004e80d47bba4"
server_key_iv: "8edd7f4020af99f6b29a19dba140dec3"
server_key_version: "encryptionkey@0"
I20260924 03:06:39.476413 19888 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260924 03:06:39.477631 19903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:39.477960 19888 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:06:39.478120 19888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "01970f3378f54466a9b144e604b5fb2a"
format_stamp: "Formatted at 2026-09-24 03:06:39 on dist-test-slave-wgsb"
server_key: "155cd8de40e5fd01ee5004e80d47bba4"
server_key_iv: "8edd7f4020af99f6b29a19dba140dec3"
server_key_version: "encryptionkey@0"
I20260924 03:06:39.478219 19888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:39.495507 19888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:39.499997 19888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:39.500188 19888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:39.500454 19888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:06:39.500794 19888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:06:39.500855 19888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:39.500902 19888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:06:39.500927 19888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:39.511082 19888 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34725
I20260924 03:06:39.511124 20016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34725 every 8 connection(s)
I20260924 03:06:39.511657 19888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:06:39.513664 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 19888
I20260924 03:06:39.514428 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:06:39.514720 392 external_mini_cluster.cc:1428] Setting key 3f76f2f46acfd72bc47a2ec2276d918e
Sep 24 03:06:39 dist-test-slave-wgsb krb5kdc[19513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:06:39.526815 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:39.513463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42735 (local address 127.0.98.62:35187)
0924 03:06:39.513521 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:39.513524 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:39.513857 (+ 333us) server_negotiation.cc:407] Connection header received
0924 03:06:39.515271 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:39.515272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:39.515291 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:39.515388 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:39.516659 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.517165 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.518322 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.518534 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.518596 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:39.522025 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:39.522027 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:39.522027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:06:39.522035 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:39.524758 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:39.525440 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:39.525442 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:39.525442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:39.525522 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:39.525907 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:39.525908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:39.525909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:39.526045 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:39.526126 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:39.526514 (+ 388us) server_negotiation.cc:299] Negotiation successful
0924 03:06:39.526627 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 03:06:39.526947 20019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:39.513469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35187 (local address 127.0.98.3:42735)
0924 03:06:39.513722 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:39.513772 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:39.514934 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:39.515549 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:39.515555 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:39.516207 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:06:39.516506 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:39.516512 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.517294 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.517295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:39.518171 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:39.518174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.518615 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.518616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:39.518733 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:39.519424 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:06:39.519433 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:06:39.521872 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:06:39.524881 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:06:39.524890 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:06:39.524892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:06:39.525340 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:06:39.525677 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:06:39.525677 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:06:39.525678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:06:39.525813 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:06:39.526206 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:06:39.526212 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:06:39.526357 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 03:06:39.526470 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 03:06:39.526623 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":111,"threads_started":1}
I20260924 03:06:39.527547 20017 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35187
I20260924 03:06:39.527726 20017 heartbeater.cc:464] Registering TS with master...
I20260924 03:06:39.527907 20017 heartbeater.cc:511] Master 127.0.98.62:35187 requested a full tablet report, sending...
I20260924 03:06:39.528458 19557 ts_manager.cc:194] Registered new tserver with Master: 01970f3378f54466a9b144e604b5fb2a (127.0.98.3:34725)
I20260924 03:06:39.528856 19557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42735
I20260924 03:06:39.538479 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:06:39.547935 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:39.539908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51258 (local address 127.0.98.62:35187)
0924 03:06:39.539984 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:39.539987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:39.540080 (+ 93us) server_negotiation.cc:407] Connection header received
0924 03:06:39.540128 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:39.540129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:39.540140 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:39.540212 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:39.540835 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.541545 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.542439 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.542654 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.542719 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:39.544499 (+ 1780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:39.544500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:39.544500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:06:39.544507 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:39.546311 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:39.546665 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:39.546666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:39.546666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:39.546734 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:39.546922 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:39.546923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:39.546924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:39.547198 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:39.547530 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:39.547767 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 03:06:39.547845 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 03:06:39.549950 19557 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51258:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:06:39.551929 19557 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:06:39.559332 20028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:39.555282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44183 (local address 127.0.0.1:37742)
0924 03:06:39.555458 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:39.555502 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:39.555570 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:39.555845 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:39.555848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:39.555850 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:39.555931 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:39.555933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.556989 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.556990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:39.558079 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:39.558082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.559012 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.559013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:39.559094 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:39.559110 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 03:06:39.559188 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 03:06:39.559241 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
I20260924 03:06:39.559351 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:39.555394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37742 (local address 127.0.98.2:44183)
0924 03:06:39.555681 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:39.555686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:39.555695 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:06:39.555738 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:39.555740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:39.555753 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:39.555841 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:39.556017 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.556886 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.558180 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.558912 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.559007 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:39.559083 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:39.559219 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 03:06:39.559265 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":103,"threads_started":1}
I20260924 03:06:39.560211 20032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:39.555285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38599 (local address 127.0.0.1:51666)
0924 03:06:39.555709 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:39.555720 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:39.555781 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:39.556366 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:39.556367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:39.556369 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:39.556433 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:39.556440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.557737 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.557738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:39.558872 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:39.558875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.559942 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.559943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:39.560029 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:39.560034 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:06:39.560086 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 03:06:39.560131 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":53,"threads_started":1}
I20260924 03:06:39.560227 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:39.555396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51666 (local address 127.0.98.1:38599)
0924 03:06:39.556135 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:39.556140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:39.556150 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:06:39.556193 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:39.556195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:39.556209 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:39.556519 (+ 310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:39.556605 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.557625 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.558994 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.559795 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.559859 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:39.559910 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:39.560115 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 03:06:39.560155 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":95,"threads_started":1}
I20260924 03:06:39.560391 19817 tablet_service.cc:1467] Processing CreateTablet for tablet 9482a743e6c74de29e19a38e0c1e0974 (DEFAULT_TABLE table=test-table [id=a8bb7edc81c146bfad20b5883d6d3158]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:06:39.560758 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:39.555512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34725 (local address 127.0.0.1:42348)
0924 03:06:39.555879 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:39.555888 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:39.555930 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:39.556319 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:39.556320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:39.556322 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:39.556371 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:39.556372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.557682 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.557682 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:39.558686 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:39.558689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.560073 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.560074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:39.560138 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:39.560143 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:06:39.560564 (+ 421us) client_negotiation.cc:240] Negotiation successful
0924 03:06:39.560676 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":35,"thread_start_us":69,"threads_started":1}
I20260924 03:06:39.560820 19817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9482a743e6c74de29e19a38e0c1e0974. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:39.560926 19683 tablet_service.cc:1467] Processing CreateTablet for tablet 9482a743e6c74de29e19a38e0c1e0974 (DEFAULT_TABLE table=test-table [id=a8bb7edc81c146bfad20b5883d6d3158]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:06:39.561074 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:39.555687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42348 (local address 127.0.98.3:34725)
0924 03:06:39.556106 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:39.556111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:39.556120 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:06:39.556155 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:39.556157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:39.556172 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:39.556243 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:39.556454 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.557545 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.558900 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.559838 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.559908 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:39.559963 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:39.560946 (+ 983us) server_negotiation.cc:299] Negotiation successful
0924 03:06:39.560996 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":130,"threads_started":1}
I20260924 03:06:39.561259 19683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9482a743e6c74de29e19a38e0c1e0974. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:39.561672 19951 tablet_service.cc:1467] Processing CreateTablet for tablet 9482a743e6c74de29e19a38e0c1e0974 (DEFAULT_TABLE table=test-table [id=a8bb7edc81c146bfad20b5883d6d3158]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:06:39.561996 19951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9482a743e6c74de29e19a38e0c1e0974. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:39.564680 20034 tablet_bootstrap.cc:492] T 9482a743e6c74de29e19a38e0c1e0974 P 39d3b539fdf5441f8bd5a43ba4dd53cd: Bootstrap starting.
I20260924 03:06:39.565563 20034 tablet_bootstrap.cc:654] T 9482a743e6c74de29e19a38e0c1e0974 P 39d3b539fdf5441f8bd5a43ba4dd53cd: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:39.566161 20035 tablet_bootstrap.cc:492] T 9482a743e6c74de29e19a38e0c1e0974 P ce0a1d07cc71410ab66b39c103392fdf: Bootstrap starting.
I20260924 03:06:39.566578 20034 log.cc:824] T 9482a743e6c74de29e19a38e0c1e0974 P 39d3b539fdf5441f8bd5a43ba4dd53cd: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:39.566907 20036 tablet_bootstrap.cc:492] T 9482a743e6c74de29e19a38e0c1e0974 P 01970f3378f54466a9b144e604b5fb2a: Bootstrap starting.
I20260924 03:06:39.567062 20035 tablet_bootstrap.cc:654] T 9482a743e6c74de29e19a38e0c1e0974 P ce0a1d07cc71410ab66b39c103392fdf: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:39.567625 20034 tablet_bootstrap.cc:492] T 9482a743e6c74de29e19a38e0c1e0974 P 39d3b539fdf5441f8bd5a43ba4dd53cd: No bootstrap required, opened a new log
I20260924 03:06:39.567674 20035 log.cc:824] T 9482a743e6c74de29e19a38e0c1e0974 P ce0a1d07cc71410ab66b39c103392fdf: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:39.567713 20034 ts_tablet_manager.cc:1397] T 9482a743e6c74de29e19a38e0c1e0974 P 39d3b539fdf5441f8bd5a43ba4dd53cd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 03:06:39.567794 20036 tablet_bootstrap.cc:654] T 9482a743e6c74de29e19a38e0c1e0974 P 01970f3378f54466a9b144e604b5fb2a: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:39.568398 20036 log.cc:824] T 9482a743e6c74de29e19a38e0c1e0974 P 01970f3378f54466a9b144e604b5fb2a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:39.568704 20035 tablet_bootstrap.cc:492] T 9482a743e6c74de29e19a38e0c1e0974 P ce0a1d07cc71410ab66b39c103392fdf: No bootstrap required, opened a new log
I20260924 03:06:39.568797 20035 ts_tablet_manager.cc:1397] T 9482a743e6c74de29e19a38e0c1e0974 P ce0a1d07cc71410ab66b39c103392fdf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 03:06:39.569221 20036 tablet_bootstrap.cc:492] T 9482a743e6c74de29e19a38e0c1e0974 P 01970f3378f54466a9b144e604b5fb2a: No bootstrap required, opened a new log
I20260924 03:06:39.569300 20036 ts_tablet_manager.cc:1397] T 9482a743e6c74de29e19a38e0c1e0974 P 01970f3378f54466a9b144e604b5fb2a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:06:39.570122 20034 raft_consensus.cc:348] T 9482a743e6c74de29e19a38e0c1e0974 P 39d3b539fdf5441f8bd5a43ba4dd53cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0a1d07cc71410ab66b39c103392fdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44183 } } peers { permanent_uuid: "39d3b539fdf5441f8bd5a43ba4dd53cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38599 } } peers { permanent_uuid: "01970f3378f54466a9b144e604b5fb2a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34725 } }
I20260924 03:06:39.570356 20034 raft_consensus.cc:374] T 9482a743e6c74de29e19a38e0c1e0974 P 39d3b539fdf5441f8bd5a43ba4dd53cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:39.570403 20034 raft_consensus.cc:729] T 9482a743e6c74de29e19a38e0c1e0974 P 39d3b539fdf5441f8bd5a43ba4dd53cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39d3b539fdf5441f8bd5a43ba4dd53cd, State: Initialized, Role: FOLLOWER
I20260924 03:06:39.570573 20034 consensus_queue.cc:260] T 9482a743e6c74de29e19a38e0c1e0974 P 39d3b539fdf5441f8bd5a43ba4dd53cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0a1d07cc71410ab66b39c103392fdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44183 } } peers { permanent_uuid: "39d3b539fdf5441f8bd5a43ba4dd53cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38599 } } peers { permanent_uuid: "01970f3378f54466a9b144e604b5fb2a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34725 } }
I20260924 03:06:39.570767 20035 raft_consensus.cc:348] T 9482a743e6c74de29e19a38e0c1e0974 P ce0a1d07cc71410ab66b39c103392fdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0a1d07cc71410ab66b39c103392fdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44183 } } peers { permanent_uuid: "39d3b539fdf5441f8bd5a43ba4dd53cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38599 } } peers { permanent_uuid: "01970f3378f54466a9b144e604b5fb2a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34725 } }
I20260924 03:06:39.570905 19749 heartbeater.cc:503] Master 127.0.98.62:35187 was elected leader, sending a full tablet report...
I20260924 03:06:39.570964 20035 raft_consensus.cc:374] T 9482a743e6c74de29e19a38e0c1e0974 P ce0a1d07cc71410ab66b39c103392fdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:39.571022 20035 raft_consensus.cc:729] T 9482a743e6c74de29e19a38e0c1e0974 P ce0a1d07cc71410ab66b39c103392fdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce0a1d07cc71410ab66b39c103392fdf, State: Initialized, Role: FOLLOWER
I20260924 03:06:39.571153 20035 consensus_queue.cc:260] T 9482a743e6c74de29e19a38e0c1e0974 P ce0a1d07cc71410ab66b39c103392fdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0a1d07cc71410ab66b39c103392fdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44183 } } peers { permanent_uuid: "39d3b539fdf5441f8bd5a43ba4dd53cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38599 } } peers { permanent_uuid: "01970f3378f54466a9b144e604b5fb2a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34725 } }
I20260924 03:06:39.571441 20035 ts_tablet_manager.cc:1428] T 9482a743e6c74de29e19a38e0c1e0974 P ce0a1d07cc71410ab66b39c103392fdf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:06:39.571566 19883 heartbeater.cc:503] Master 127.0.98.62:35187 was elected leader, sending a full tablet report...
I20260924 03:06:39.571534 20036 raft_consensus.cc:348] T 9482a743e6c74de29e19a38e0c1e0974 P 01970f3378f54466a9b144e604b5fb2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0a1d07cc71410ab66b39c103392fdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44183 } } peers { permanent_uuid: "39d3b539fdf5441f8bd5a43ba4dd53cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38599 } } peers { permanent_uuid: "01970f3378f54466a9b144e604b5fb2a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34725 } }
I20260924 03:06:39.570896 20034 ts_tablet_manager.cc:1428] T 9482a743e6c74de29e19a38e0c1e0974 P 39d3b539fdf5441f8bd5a43ba4dd53cd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:06:39.571740 20036 raft_consensus.cc:374] T 9482a743e6c74de29e19a38e0c1e0974 P 01970f3378f54466a9b144e604b5fb2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:39.571784 20036 raft_consensus.cc:729] T 9482a743e6c74de29e19a38e0c1e0974 P 01970f3378f54466a9b144e604b5fb2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01970f3378f54466a9b144e604b5fb2a, State: Initialized, Role: FOLLOWER
I20260924 03:06:39.571900 20036 consensus_queue.cc:260] T 9482a743e6c74de29e19a38e0c1e0974 P 01970f3378f54466a9b144e604b5fb2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0a1d07cc71410ab66b39c103392fdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44183 } } peers { permanent_uuid: "39d3b539fdf5441f8bd5a43ba4dd53cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38599 } } peers { permanent_uuid: "01970f3378f54466a9b144e604b5fb2a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34725 } }
I20260924 03:06:39.572203 20036 ts_tablet_manager.cc:1428] T 9482a743e6c74de29e19a38e0c1e0974 P 01970f3378f54466a9b144e604b5fb2a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:06:39.572247 20017 heartbeater.cc:503] Master 127.0.98.62:35187 was elected leader, sending a full tablet report...
W20260924 03:06:39.576021 19884 tablet.cc:2369] T 9482a743e6c74de29e19a38e0c1e0974 P ce0a1d07cc71410ab66b39c103392fdf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:06:39.668897 19750 tablet.cc:2369] T 9482a743e6c74de29e19a38e0c1e0974 P 39d3b539fdf5441f8bd5a43ba4dd53cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:06:39.677517 20040 raft_consensus.cc:482] T 9482a743e6c74de29e19a38e0c1e0974 P 39d3b539fdf5441f8bd5a43ba4dd53cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:06:39.678170 20040 raft_consensus.cc:504] T 9482a743e6c74de29e19a38e0c1e0974 P 39d3b539fdf5441f8bd5a43ba4dd53cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0a1d07cc71410ab66b39c103392fdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44183 } } peers { permanent_uuid: "39d3b539fdf5441f8bd5a43ba4dd53cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38599 } } peers { permanent_uuid: "01970f3378f54466a9b144e604b5fb2a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34725 } }
I20260924 03:06:39.678678 20040 leader_election.cc:288] T 9482a743e6c74de29e19a38e0c1e0974 P 39d3b539fdf5441f8bd5a43ba4dd53cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce0a1d07cc71410ab66b39c103392fdf (127.0.98.2:44183), 01970f3378f54466a9b144e604b5fb2a (127.0.98.3:34725)
I20260924 03:06:39.685547 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:39.679583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50091 (local address 127.0.98.3:34725)
0924 03:06:39.679679 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:39.679682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:39.679818 (+ 136us) server_negotiation.cc:407] Connection header received
0924 03:06:39.680236 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:39.680237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:39.680250 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:39.680330 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:39.680809 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.681516 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.684075 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.684815 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.684896 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:39.684908 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:39.685388 (+ 480us) server_negotiation.cc:299] Negotiation successful
0924 03:06:39.685465 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 03:06:39.685547 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:39.679502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44947 (local address 127.0.98.2:44183)
0924 03:06:39.679579 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:39.679581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:39.679630 (+ 49us) server_negotiation.cc:407] Connection header received
0924 03:06:39.679690 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:39.679691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:39.679710 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:39.679764 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:39.680234 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.681517 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.684075 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.684752 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.684867 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:39.684887 (+ 20us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:39.685388 (+ 501us) server_negotiation.cc:299] Negotiation successful
0924 03:06:39.685456 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 03:06:39.685531 19751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:39.679257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44183 (local address 127.0.98.1:44947)
0924 03:06:39.679374 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:39.679391 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:39.679476 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:39.679873 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:39.679874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:39.679877 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:39.679975 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:39.679977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.681721 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.681722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:39.683625 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:39.683633 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.685084 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.685085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:39.685190 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:39.685200 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 03:06:39.685294 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 03:06:39.685372 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":66}
I20260924 03:06:39.685537 20043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:39.679257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34725 (local address 127.0.98.1:50091)
0924 03:06:39.679565 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:39.679578 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:39.679875 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:39.680401 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:39.680402 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:39.680404 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:39.680493 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:39.680494 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.681721 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.681722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:39.683633 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:39.683635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.685080 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.685083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:39.685190 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:39.685201 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 03:06:39.685350 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 03:06:39.685397 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":306,"thread_start_us":144,"threads_started":1}
I20260924 03:06:39.686600 19837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9482a743e6c74de29e19a38e0c1e0974" candidate_uuid: "39d3b539fdf5441f8bd5a43ba4dd53cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce0a1d07cc71410ab66b39c103392fdf" is_pre_election: true
I20260924 03:06:39.686807 19971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9482a743e6c74de29e19a38e0c1e0974" candidate_uuid: "39d3b539fdf5441f8bd5a43ba4dd53cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01970f3378f54466a9b144e604b5fb2a" is_pre_election: true
I20260924 03:06:39.686866 19837 raft_consensus.cc:2454] T 9482a743e6c74de29e19a38e0c1e0974 P ce0a1d07cc71410ab66b39c103392fdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39d3b539fdf5441f8bd5a43ba4dd53cd in term 0.
I20260924 03:06:39.686949 19971 raft_consensus.cc:2454] T 9482a743e6c74de29e19a38e0c1e0974 P 01970f3378f54466a9b144e604b5fb2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39d3b539fdf5441f8bd5a43ba4dd53cd in term 0.
I20260924 03:06:39.687170 19637 leader_election.cc:302] T 9482a743e6c74de29e19a38e0c1e0974 P 39d3b539fdf5441f8bd5a43ba4dd53cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39d3b539fdf5441f8bd5a43ba4dd53cd, ce0a1d07cc71410ab66b39c103392fdf; no voters:
I20260924 03:06:39.687318 20040 raft_consensus.cc:2781] T 9482a743e6c74de29e19a38e0c1e0974 P 39d3b539fdf5441f8bd5a43ba4dd53cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:06:39.687386 20040 raft_consensus.cc:482] T 9482a743e6c74de29e19a38e0c1e0974 P 39d3b539fdf5441f8bd5a43ba4dd53cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:06:39.687403 20040 raft_consensus.cc:3037] T 9482a743e6c74de29e19a38e0c1e0974 P 39d3b539fdf5441f8bd5a43ba4dd53cd [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:39.688444 20040 raft_consensus.cc:504] T 9482a743e6c74de29e19a38e0c1e0974 P 39d3b539fdf5441f8bd5a43ba4dd53cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0a1d07cc71410ab66b39c103392fdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44183 } } peers { permanent_uuid: "39d3b539fdf5441f8bd5a43ba4dd53cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38599 } } peers { permanent_uuid: "01970f3378f54466a9b144e604b5fb2a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34725 } }
I20260924 03:06:39.688828 20040 leader_election.cc:288] T 9482a743e6c74de29e19a38e0c1e0974 P 39d3b539fdf5441f8bd5a43ba4dd53cd [CANDIDATE]: Term 1 election: Requested vote from peers ce0a1d07cc71410ab66b39c103392fdf (127.0.98.2:44183), 01970f3378f54466a9b144e604b5fb2a (127.0.98.3:34725)
I20260924 03:06:39.689062 19837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9482a743e6c74de29e19a38e0c1e0974" candidate_uuid: "39d3b539fdf5441f8bd5a43ba4dd53cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce0a1d07cc71410ab66b39c103392fdf"
I20260924 03:06:39.689235 19971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9482a743e6c74de29e19a38e0c1e0974" candidate_uuid: "39d3b539fdf5441f8bd5a43ba4dd53cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01970f3378f54466a9b144e604b5fb2a"
I20260924 03:06:39.689364 19837 raft_consensus.cc:3037] T 9482a743e6c74de29e19a38e0c1e0974 P ce0a1d07cc71410ab66b39c103392fdf [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:39.689356 19971 raft_consensus.cc:3037] T 9482a743e6c74de29e19a38e0c1e0974 P 01970f3378f54466a9b144e604b5fb2a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:39.690289 19837 raft_consensus.cc:2454] T 9482a743e6c74de29e19a38e0c1e0974 P ce0a1d07cc71410ab66b39c103392fdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39d3b539fdf5441f8bd5a43ba4dd53cd in term 1.
I20260924 03:06:39.690420 19971 raft_consensus.cc:2454] T 9482a743e6c74de29e19a38e0c1e0974 P 01970f3378f54466a9b144e604b5fb2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39d3b539fdf5441f8bd5a43ba4dd53cd in term 1.
I20260924 03:06:39.690747 19637 leader_election.cc:302] T 9482a743e6c74de29e19a38e0c1e0974 P 39d3b539fdf5441f8bd5a43ba4dd53cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39d3b539fdf5441f8bd5a43ba4dd53cd, ce0a1d07cc71410ab66b39c103392fdf; no voters:
I20260924 03:06:39.690869 20040 raft_consensus.cc:2781] T 9482a743e6c74de29e19a38e0c1e0974 P 39d3b539fdf5441f8bd5a43ba4dd53cd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:06:39.691082 20040 raft_consensus.cc:686] T 9482a743e6c74de29e19a38e0c1e0974 P 39d3b539fdf5441f8bd5a43ba4dd53cd [term 1 LEADER]: Becoming Leader. State: Replica: 39d3b539fdf5441f8bd5a43ba4dd53cd, State: Running, Role: LEADER
I20260924 03:06:39.691169 20040 consensus_queue.cc:237] T 9482a743e6c74de29e19a38e0c1e0974 P 39d3b539fdf5441f8bd5a43ba4dd53cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0a1d07cc71410ab66b39c103392fdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44183 } } peers { permanent_uuid: "39d3b539fdf5441f8bd5a43ba4dd53cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38599 } } peers { permanent_uuid: "01970f3378f54466a9b144e604b5fb2a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34725 } }
I20260924 03:06:39.692437 19558 catalog_manager.cc:5495] T 9482a743e6c74de29e19a38e0c1e0974 P 39d3b539fdf5441f8bd5a43ba4dd53cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 39d3b539fdf5441f8bd5a43ba4dd53cd (127.0.98.1). New cstate: current_term: 1 leader_uuid: "39d3b539fdf5441f8bd5a43ba4dd53cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0a1d07cc71410ab66b39c103392fdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39d3b539fdf5441f8bd5a43ba4dd53cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01970f3378f54466a9b144e604b5fb2a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34725 } health_report { overall_health: UNKNOWN } } }
I20260924 03:06:39.711990 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:39.708234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51682 (local address 127.0.98.1:38599)
0924 03:06:39.708534 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:39.708537 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:39.708548 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:06:39.708614 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:39.708615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:39.708646 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:39.708991 (+ 345us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:06:39.709314 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.709874 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.710840 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.711051 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.711120 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:39.711397 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:06:39.711530 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:06:39.711639 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:39.711852 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 03:06:39.711885 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263}
I20260924 03:06:39.714954 19837 raft_consensus.cc:1261] T 9482a743e6c74de29e19a38e0c1e0974 P ce0a1d07cc71410ab66b39c103392fdf [term 1 FOLLOWER]: Refusing update from remote peer 39d3b539fdf5441f8bd5a43ba4dd53cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:06:39.715211 19971 raft_consensus.cc:1261] T 9482a743e6c74de29e19a38e0c1e0974 P 01970f3378f54466a9b144e604b5fb2a [term 1 FOLLOWER]: Refusing update from remote peer 39d3b539fdf5441f8bd5a43ba4dd53cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:06:39.715730 20040 consensus_queue.cc:1035] T 9482a743e6c74de29e19a38e0c1e0974 P 39d3b539fdf5441f8bd5a43ba4dd53cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce0a1d07cc71410ab66b39c103392fdf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:06:39.715992 20044 consensus_queue.cc:1035] T 9482a743e6c74de29e19a38e0c1e0974 P 39d3b539fdf5441f8bd5a43ba4dd53cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "01970f3378f54466a9b144e604b5fb2a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:06:39.721071 20048 mvcc.cc:205] Tried to move back new op lower bound from 7332737842027851776 to 7332737841935761408. Current Snapshot: MvccSnapshot[applied={T|T < 7332737842027851776}]
I20260924 03:06:39.725970 19558 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:06:39.726142 19558 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:06:39.727890 19558 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ded570286a8464e86c9c3136fbd0924: Sending DeleteTablet for 3 replicas of tablet 9482a743e6c74de29e19a38e0c1e0974
I20260924 03:06:39.728437 19683 tablet_service.cc:1514] Processing DeleteTablet for tablet 9482a743e6c74de29e19a38e0c1e0974 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:06:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51666
I20260924 03:06:39.728453 19817 tablet_service.cc:1514] Processing DeleteTablet for tablet 9482a743e6c74de29e19a38e0c1e0974 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:06:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37742
I20260924 03:06:39.728796 20059 tablet_replica.cc:330] T 9482a743e6c74de29e19a38e0c1e0974 P ce0a1d07cc71410ab66b39c103392fdf: stopping tablet replica
I20260924 03:06:39.728813 19951 tablet_service.cc:1514] Processing DeleteTablet for tablet 9482a743e6c74de29e19a38e0c1e0974 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:06:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42348
I20260924 03:06:39.728931 20059 raft_consensus.cc:2229] T 9482a743e6c74de29e19a38e0c1e0974 P ce0a1d07cc71410ab66b39c103392fdf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:06:39.729069 20059 raft_consensus.cc:2258] T 9482a743e6c74de29e19a38e0c1e0974 P ce0a1d07cc71410ab66b39c103392fdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:06:39.729709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 19620
I20260924 03:06:39.731570 20059 ts_tablet_manager.cc:1905] T 9482a743e6c74de29e19a38e0c1e0974 P ce0a1d07cc71410ab66b39c103392fdf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:06:39.735354 20059 ts_tablet_manager.cc:1918] T 9482a743e6c74de29e19a38e0c1e0974 P ce0a1d07cc71410ab66b39c103392fdf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:06:39.735479 20059 log.cc:1196] T 9482a743e6c74de29e19a38e0c1e0974 P ce0a1d07cc71410ab66b39c103392fdf: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-1/wal/wals/9482a743e6c74de29e19a38e0c1e0974
I20260924 03:06:39.735858 20059 ts_tablet_manager.cc:1939] T 9482a743e6c74de29e19a38e0c1e0974 P ce0a1d07cc71410ab66b39c103392fdf: Deleting consensus metadata
I20260924 03:06:39.735872 20060 tablet_replica.cc:330] T 9482a743e6c74de29e19a38e0c1e0974 P 01970f3378f54466a9b144e604b5fb2a: stopping tablet replica
I20260924 03:06:39.736116 20060 raft_consensus.cc:2229] T 9482a743e6c74de29e19a38e0c1e0974 P 01970f3378f54466a9b144e604b5fb2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:06:39.736738 20060 raft_consensus.cc:2258] T 9482a743e6c74de29e19a38e0c1e0974 P 01970f3378f54466a9b144e604b5fb2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:06:39.736804 19543 catalog_manager.cc:4841] TS ce0a1d07cc71410ab66b39c103392fdf (127.0.98.2:44183): tablet 9482a743e6c74de29e19a38e0c1e0974 (table test-table [id=a8bb7edc81c146bfad20b5883d6d3158]) successfully deleted
I20260924 03:06:39.737461 20060 ts_tablet_manager.cc:1905] T 9482a743e6c74de29e19a38e0c1e0974 P 01970f3378f54466a9b144e604b5fb2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:06:39.743057 20060 ts_tablet_manager.cc:1918] T 9482a743e6c74de29e19a38e0c1e0974 P 01970f3378f54466a9b144e604b5fb2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:06:39.744185 20060 log.cc:1196] T 9482a743e6c74de29e19a38e0c1e0974 P 01970f3378f54466a9b144e604b5fb2a: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218948286999-392-0/minicluster-data/ts-2/wal/wals/9482a743e6c74de29e19a38e0c1e0974
I20260924 03:06:39.744697 20060 ts_tablet_manager.cc:1939] T 9482a743e6c74de29e19a38e0c1e0974 P 01970f3378f54466a9b144e604b5fb2a: Deleting consensus metadata
W20260924 03:06:39.744768 19545 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38599 (error 108)
I20260924 03:06:39.745843 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 19754
I20260924 03:06:39.746243 19545 catalog_manager.cc:4841] TS 01970f3378f54466a9b144e604b5fb2a (127.0.98.3:34725): tablet 9482a743e6c74de29e19a38e0c1e0974 (table test-table [id=a8bb7edc81c146bfad20b5883d6d3158]) successfully deleted
I20260924 03:06:39.746511 20032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:39.746193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38599 (local address 127.0.0.1:51690)
0924 03:06:39.746434 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:39.746472 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215}
W20260924 03:06:39.746671 19545 catalog_manager.cc:4584] TS 39d3b539fdf5441f8bd5a43ba4dd53cd (127.0.98.1:38599): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9482a743e6c74de29e19a38e0c1e0974: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38599: connect: Connection refused (error 111)
I20260924 03:06:39.754570 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 19888
I20260924 03:06:39.761181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 19526
2026-09-24T03:06:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2113 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T03:06:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:06:39Z Disabled control of system clock
I20260924 03:06:39.791705 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44257
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:06:39.876564 20065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:39.876812 20065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:39.876835 20065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:39.878212 20065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:06:39.878278 20065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:39.878293 20065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:39.878309 20065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:06:39.878324 20065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:06:39.879813 20065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44257
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:39.880106 20065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:39.880434 20065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:39.885988 20070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:39.885967 20073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:39.886049 20065 server_base.cc:1034] running on GCE node
W20260924 03:06:39.885952 20071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:39.886706 20065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:39.887042 20065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:39.888229 20065 hybrid_clock.cc:648] HybridClock initialized: now 1790219199888206 us; error 41 us; skew 500 ppm
I20260924 03:06:39.889633 20065 webserver.cc:460] Webserver started at http://127.0.98.62:38309/ using document root <none> and password file <none>
I20260924 03:06:39.889920 20065 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:39.889997 20065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:39.890158 20065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:39.891258 20065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "991258a83f434e4d8fb536239bbead4c"
format_stamp: "Formatted at 2026-09-24 03:06:39 on dist-test-slave-wgsb"
server_key: "8b183fa5c9be97c9562e3f96455859c3"
server_key_iv: "51ce22f35f4b2d83069634a6fa501f97"
server_key_version: "encryptionkey@0"
I20260924 03:06:39.891688 20065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "991258a83f434e4d8fb536239bbead4c"
format_stamp: "Formatted at 2026-09-24 03:06:39 on dist-test-slave-wgsb"
server_key: "8b183fa5c9be97c9562e3f96455859c3"
server_key_iv: "51ce22f35f4b2d83069634a6fa501f97"
server_key_version: "encryptionkey@0"
I20260924 03:06:39.893174 20065 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:06:39.894023 20079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:39.894260 20065 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:06:39.894358 20065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "991258a83f434e4d8fb536239bbead4c"
format_stamp: "Formatted at 2026-09-24 03:06:39 on dist-test-slave-wgsb"
server_key: "8b183fa5c9be97c9562e3f96455859c3"
server_key_iv: "51ce22f35f4b2d83069634a6fa501f97"
server_key_version: "encryptionkey@0"
I20260924 03:06:39.894445 20065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:39.918504 20065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:39.918929 20065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:39.919097 20065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:39.923807 20065 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44257
I20260924 03:06:39.923959 20131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44257 every 8 connection(s)
I20260924 03:06:39.924260 20065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:06:39.925323 20132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:39.926924 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 20065
I20260924 03:06:39.927052 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:06:39.927277 392 external_mini_cluster.cc:1428] Setting key a132158fe394bde37c0415bc6f7273e9
I20260924 03:06:39.928100 20132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 991258a83f434e4d8fb536239bbead4c: Bootstrap starting.
I20260924 03:06:39.928690 20132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 991258a83f434e4d8fb536239bbead4c: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:39.929034 20132 log.cc:824] T 00000000000000000000000000000000 P 991258a83f434e4d8fb536239bbead4c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:39.929733 20132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 991258a83f434e4d8fb536239bbead4c: No bootstrap required, opened a new log
I20260924 03:06:39.932123 20132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 991258a83f434e4d8fb536239bbead4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "991258a83f434e4d8fb536239bbead4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44257 } }
I20260924 03:06:39.932279 20132 raft_consensus.cc:374] T 00000000000000000000000000000000 P 991258a83f434e4d8fb536239bbead4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:39.932302 20132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 991258a83f434e4d8fb536239bbead4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 991258a83f434e4d8fb536239bbead4c, State: Initialized, Role: FOLLOWER
I20260924 03:06:39.932426 20132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 991258a83f434e4d8fb536239bbead4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "991258a83f434e4d8fb536239bbead4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44257 } }
I20260924 03:06:39.932520 20132 raft_consensus.cc:388] T 00000000000000000000000000000000 P 991258a83f434e4d8fb536239bbead4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:06:39.932571 20132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 991258a83f434e4d8fb536239bbead4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:06:39.932621 20132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 991258a83f434e4d8fb536239bbead4c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:39.933182 20132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 991258a83f434e4d8fb536239bbead4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "991258a83f434e4d8fb536239bbead4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44257 } }
I20260924 03:06:39.933300 20132 leader_election.cc:302] T 00000000000000000000000000000000 P 991258a83f434e4d8fb536239bbead4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 991258a83f434e4d8fb536239bbead4c; no voters:
I20260924 03:06:39.933459 20135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:39.927854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46464 (local address 127.0.98.62:44257)
0924 03:06:39.928083 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:39.928093 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:39.928130 (+ 37us) server_negotiation.cc:407] Connection header received
0924 03:06:39.928584 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:39.928591 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:39.928656 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:39.928805 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:39.929133 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.930131 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.931246 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:39.931733 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:39.931848 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:39.932683 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:39.932687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:39.932688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:39.932874 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:39.932906 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:39.932918 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:39.933019 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:39.933099 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 03:06:39.933182 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
I20260924 03:06:39.933487 20132 leader_election.cc:288] T 00000000000000000000000000000000 P 991258a83f434e4d8fb536239bbead4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:06:39.933540 20137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 991258a83f434e4d8fb536239bbead4c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:06:39.933737 20137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 991258a83f434e4d8fb536239bbead4c [term 1 LEADER]: Becoming Leader. State: Replica: 991258a83f434e4d8fb536239bbead4c, State: Running, Role: LEADER
I20260924 03:06:39.933884 20137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 991258a83f434e4d8fb536239bbead4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "991258a83f434e4d8fb536239bbead4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44257 } }
I20260924 03:06:39.933785 20132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 991258a83f434e4d8fb536239bbead4c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:06:39.935066 20138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 991258a83f434e4d8fb536239bbead4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "991258a83f434e4d8fb536239bbead4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "991258a83f434e4d8fb536239bbead4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44257 } } }
I20260924 03:06:39.935153 20138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 991258a83f434e4d8fb536239bbead4c [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:39.935279 20139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 991258a83f434e4d8fb536239bbead4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 991258a83f434e4d8fb536239bbead4c. Latest consensus state: current_term: 1 leader_uuid: "991258a83f434e4d8fb536239bbead4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "991258a83f434e4d8fb536239bbead4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44257 } } }
I20260924 03:06:39.935365 20153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:06:39.935374 20139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 991258a83f434e4d8fb536239bbead4c [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:39.936081 20153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:06:39.937456 20153 catalog_manager.cc:1347] Generated new cluster ID: 1933ad4cc3ee4f7db168f9b77bd1e3aa
I20260924 03:06:39.937510 20153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:06:39.944802 20153 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:06:39.945508 20153 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:06:39.948298 20153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 991258a83f434e4d8fb536239bbead4c: Generated new TSK 0
I20260924 03:06:39.948467 20153 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:06:39.952157 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44257
--builtin_ntp_servers=127.0.98.20:46151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:06:40.037322 20156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:40.037551 20156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:40.037590 20156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:40.038805 20156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:40.038844 20156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:40.038872 20156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:06:40.040304 20156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:40.040547 20156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:40.040879 20156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:40.046846 20164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:40.046876 20162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:40.046942 20161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:40.047135 20156 server_base.cc:1034] running on GCE node
I20260924 03:06:40.047398 20156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:40.047683 20156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:40.048854 20156 hybrid_clock.cc:648] HybridClock initialized: now 1790219200048835 us; error 33 us; skew 500 ppm
I20260924 03:06:40.050127 20156 webserver.cc:460] Webserver started at http://127.0.98.1:46497/ using document root <none> and password file <none>
I20260924 03:06:40.050372 20156 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:40.050439 20156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:40.050575 20156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:40.051751 20156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-0/data/instance:
uuid: "e554acf2059047feb4f6dd7847a119b8"
format_stamp: "Formatted at 2026-09-24 03:06:40 on dist-test-slave-wgsb"
server_key: "e5601a975886192b4d56e086488eef68"
server_key_iv: "fe936031d43df352fedaf180b8788283"
server_key_version: "encryptionkey@0"
I20260924 03:06:40.052105 20156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e554acf2059047feb4f6dd7847a119b8"
format_stamp: "Formatted at 2026-09-24 03:06:40 on dist-test-slave-wgsb"
server_key: "e5601a975886192b4d56e086488eef68"
server_key_iv: "fe936031d43df352fedaf180b8788283"
server_key_version: "encryptionkey@0"
I20260924 03:06:40.053422 20156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:06:40.054205 20170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:40.054379 20156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:06:40.054477 20156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-0/wal
uuid: "e554acf2059047feb4f6dd7847a119b8"
format_stamp: "Formatted at 2026-09-24 03:06:40 on dist-test-slave-wgsb"
server_key: "e5601a975886192b4d56e086488eef68"
server_key_iv: "fe936031d43df352fedaf180b8788283"
server_key_version: "encryptionkey@0"
I20260924 03:06:40.054561 20156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:40.077560 20156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:40.077965 20156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:40.078137 20156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:40.078383 20156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:06:40.078809 20156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:06:40.078856 20156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:40.078887 20156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:06:40.078905 20156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:40.086772 20156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40881
I20260924 03:06:40.086813 20283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40881 every 8 connection(s)
I20260924 03:06:40.087190 20156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:06:40.088423 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 20156
I20260924 03:06:40.088541 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:06:40.088742 392 external_mini_cluster.cc:1428] Setting key cf4a30bd72ac3301677ccaac62a4c542
I20260924 03:06:40.091988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44257
--builtin_ntp_servers=127.0.98.20:46151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:06:40.095216 20135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:40.088026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47385 (local address 127.0.98.62:44257)
0924 03:06:40.088070 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:40.088072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:40.088358 (+ 286us) server_negotiation.cc:407] Connection header received
0924 03:06:40.089118 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:40.089119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:40.089136 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:40.089178 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:40.090236 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.090725 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.092341 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.092655 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.092739 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:40.094887 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:40.094889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:40.094889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:40.094895 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:40.094913 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:40.094918 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:40.094978 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:40.095068 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 03:06:40.095098 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:06:40.095454 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:40.088020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44257 (local address 127.0.98.1:47385)
0924 03:06:40.088223 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:40.088271 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:40.088997 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:40.089357 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:40.089360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:40.089368 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:06:40.090109 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:40.090117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.090828 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.090829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:40.092215 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:40.092218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.092756 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.092757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:40.092925 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:40.094717 (+ 1792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:06:40.094717 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:06:40.094759 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:06:40.094790 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:06:40.094992 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:06:40.094998 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:06:40.095000 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:06:40.095070 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 03:06:40.095185 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
I20260924 03:06:40.096076 20284 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44257
I20260924 03:06:40.096196 20284 heartbeater.cc:464] Registering TS with master...
I20260924 03:06:40.096382 20284 heartbeater.cc:511] Master 127.0.98.62:44257 requested a full tablet report, sending...
I20260924 03:06:40.096973 20096 ts_manager.cc:194] Registered new tserver with Master: e554acf2059047feb4f6dd7847a119b8 (127.0.98.1:40881)
I20260924 03:06:40.097596 20096 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:47385
W20260924 03:06:40.185595 20287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:40.185824 20287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:40.185847 20287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:40.186995 20287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:40.187036 20287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:40.187062 20287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:06:40.188474 20287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:40.188719 20287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:40.189026 20287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:40.194521 20295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:40.194521 20292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:40.194535 20293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:40.194550 20287 server_base.cc:1034] running on GCE node
I20260924 03:06:40.194999 20287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:40.195287 20287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:40.196456 20287 hybrid_clock.cc:648] HybridClock initialized: now 1790219200196431 us; error 40 us; skew 500 ppm
I20260924 03:06:40.197902 20287 webserver.cc:460] Webserver started at http://127.0.98.2:33739/ using document root <none> and password file <none>
I20260924 03:06:40.198210 20287 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:40.198333 20287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:40.198489 20287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:40.199580 20287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-1/data/instance:
uuid: "a7a1d065bb034d61993b9f609a482b51"
format_stamp: "Formatted at 2026-09-24 03:06:40 on dist-test-slave-wgsb"
server_key: "bbab225e381b5a8918282c70e3eaf95a"
server_key_iv: "809cf04e08708bef80a1ad3115756d1d"
server_key_version: "encryptionkey@0"
I20260924 03:06:40.200022 20287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a7a1d065bb034d61993b9f609a482b51"
format_stamp: "Formatted at 2026-09-24 03:06:40 on dist-test-slave-wgsb"
server_key: "bbab225e381b5a8918282c70e3eaf95a"
server_key_iv: "809cf04e08708bef80a1ad3115756d1d"
server_key_version: "encryptionkey@0"
I20260924 03:06:40.201412 20287 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:06:40.202188 20301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:40.202373 20287 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:06:40.202474 20287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-1/wal
uuid: "a7a1d065bb034d61993b9f609a482b51"
format_stamp: "Formatted at 2026-09-24 03:06:40 on dist-test-slave-wgsb"
server_key: "bbab225e381b5a8918282c70e3eaf95a"
server_key_iv: "809cf04e08708bef80a1ad3115756d1d"
server_key_version: "encryptionkey@0"
I20260924 03:06:40.202564 20287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:40.214781 20287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:40.215119 20287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:40.215276 20287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:40.215498 20287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:06:40.215927 20287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:06:40.215987 20287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:40.216027 20287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:06:40.216056 20287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:40.222064 20287 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33573
I20260924 03:06:40.222167 20414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33573 every 8 connection(s)
I20260924 03:06:40.222499 20287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:06:40.227921 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 20287
I20260924 03:06:40.228071 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:06:40.228302 392 external_mini_cluster.cc:1428] Setting key 91810874123170a33202065ac9c0d370
I20260924 03:06:40.228754 20135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:40.223357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37731 (local address 127.0.98.62:44257)
0924 03:06:40.223411 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:40.223413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:40.223666 (+ 253us) server_negotiation.cc:407] Connection header received
0924 03:06:40.224298 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:40.224300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:40.224312 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:40.224368 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:40.225072 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.225583 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.226599 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.226818 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.226895 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:40.228402 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:40.228405 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:40.228405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:40.228411 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:40.228429 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:40.228434 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:40.228517 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:40.228629 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 03:06:40.228663 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 03:06:40.229055 20417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:40.223340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44257 (local address 127.0.98.2:37731)
0924 03:06:40.223523 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:40.223558 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:40.224099 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:40.224446 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:40.224450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:40.224457 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:06:40.224967 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:40.224972 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.225695 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.225697 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:40.226493 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:40.226495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.226904 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.226905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:40.227041 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:40.228173 (+ 1132us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:06:40.228173 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:06:40.228226 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:06:40.228270 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:06:40.228517 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:06:40.228524 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:06:40.228528 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:06:40.228630 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 03:06:40.228762 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":100,"threads_started":1}
I20260924 03:06:40.229490 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44257
--builtin_ntp_servers=127.0.98.20:46151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:06:40.229825 20415 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44257
I20260924 03:06:40.229943 20415 heartbeater.cc:464] Registering TS with master...
I20260924 03:06:40.230167 20415 heartbeater.cc:511] Master 127.0.98.62:44257 requested a full tablet report, sending...
I20260924 03:06:40.230649 20096 ts_manager.cc:194] Registered new tserver with Master: a7a1d065bb034d61993b9f609a482b51 (127.0.98.2:33573)
I20260924 03:06:40.231065 20096 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:37731
W20260924 03:06:40.317601 20418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:40.317839 20418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:40.317865 20418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:40.318998 20418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:40.319051 20418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:40.319079 20418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:06:40.320484 20418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:40.320717 20418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:40.320995 20418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:40.326618 20426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:40.326618 20423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:40.326679 20424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:40.327533 20418 server_base.cc:1034] running on GCE node
I20260924 03:06:40.327831 20418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:40.328063 20418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:40.329221 20418 hybrid_clock.cc:648] HybridClock initialized: now 1790219200329200 us; error 31 us; skew 500 ppm
I20260924 03:06:40.330634 20418 webserver.cc:460] Webserver started at http://127.0.98.3:42741/ using document root <none> and password file <none>
I20260924 03:06:40.331168 20418 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:40.331280 20418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:40.331444 20418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:40.332715 20418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-2/data/instance:
uuid: "017fca85f756452095ef7bef49ccd4de"
format_stamp: "Formatted at 2026-09-24 03:06:40 on dist-test-slave-wgsb"
server_key: "b115cf00a693c1c29730d529d21e985e"
server_key_iv: "93ffbce80e48bbe86314e0e95cfd66e6"
server_key_version: "encryptionkey@0"
I20260924 03:06:40.333091 20418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-2/wal/instance:
uuid: "017fca85f756452095ef7bef49ccd4de"
format_stamp: "Formatted at 2026-09-24 03:06:40 on dist-test-slave-wgsb"
server_key: "b115cf00a693c1c29730d529d21e985e"
server_key_iv: "93ffbce80e48bbe86314e0e95cfd66e6"
server_key_version: "encryptionkey@0"
I20260924 03:06:40.334645 20418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:06:40.335561 20432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:40.335866 20418 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:06:40.335963 20418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-2/wal
uuid: "017fca85f756452095ef7bef49ccd4de"
format_stamp: "Formatted at 2026-09-24 03:06:40 on dist-test-slave-wgsb"
server_key: "b115cf00a693c1c29730d529d21e985e"
server_key_iv: "93ffbce80e48bbe86314e0e95cfd66e6"
server_key_version: "encryptionkey@0"
I20260924 03:06:40.336062 20418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:40.359272 20418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:40.359724 20418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:40.359928 20418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:40.360196 20418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:06:40.360561 20418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:06:40.360634 20418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:40.360683 20418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:06:40.360819 20418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:40.368396 20418 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36069
I20260924 03:06:40.368443 20545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36069 every 8 connection(s)
I20260924 03:06:40.368880 20418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:06:40.375048 20135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:40.369964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33299 (local address 127.0.98.62:44257)
0924 03:06:40.370115 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:40.370118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:40.370248 (+ 130us) server_negotiation.cc:407] Connection header received
0924 03:06:40.370813 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:40.370814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:40.370826 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:40.370899 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:40.371828 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.372392 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.373594 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.373792 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.373854 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:40.374787 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:40.374789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:40.374789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:40.374794 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:40.374811 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:40.374815 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:40.374860 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:40.374948 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 03:06:40.374978 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260924 03:06:40.375346 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:40.369854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44257 (local address 127.0.98.3:33299)
0924 03:06:40.370098 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:40.370142 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:40.370659 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:40.371053 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:40.371058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:40.371068 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:06:40.371688 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:40.371694 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.372561 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.372562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:40.373479 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:40.373482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.373889 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.373890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:40.374006 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:40.374634 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:06:40.374634 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:06:40.374673 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:06:40.374699 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:06:40.374873 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:06:40.374877 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:06:40.374880 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:06:40.374933 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 03:06:40.375018 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":139,"threads_started":1}
I20260924 03:06:40.375941 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 20418
I20260924 03:06:40.375967 20546 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44257
I20260924 03:06:40.376070 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:06:40.376086 20546 heartbeater.cc:464] Registering TS with master...
I20260924 03:06:40.376289 392 external_mini_cluster.cc:1428] Setting key 9b3fe52a8cb9ebe8bd1aff03f834b274
I20260924 03:06:40.376296 20546 heartbeater.cc:511] Master 127.0.98.62:44257 requested a full tablet report, sending...
I20260924 03:06:40.376767 20095 ts_manager.cc:194] Registered new tserver with Master: 017fca85f756452095ef7bef49ccd4de (127.0.98.3:36069)
I20260924 03:06:40.377249 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:06:40.377240 20095 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:33299
I20260924 03:06:40.383018 20135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:40.378953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46470 (local address 127.0.98.62:44257)
0924 03:06:40.379133 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:40.379135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:40.379145 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:06:40.379196 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:40.379196 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:40.379206 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:40.379259 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:40.379497 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.380046 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.381638 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.381869 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.381950 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:40.382788 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:40.382789 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:40.382790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:40.382794 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:40.382807 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:40.382811 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:40.382862 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:40.382915 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 03:06:40.382936 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260924 03:06:40.385609 20095 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:06:40.387656 20095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:06:40.394784 20557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:40.390457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49690 (local address 127.0.98.3:36069)
0924 03:06:40.390970 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:40.390974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:40.390985 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:06:40.391037 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:40.391040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:40.391091 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:40.391196 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:40.391492 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.392431 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.393658 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.394322 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.394407 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:40.394505 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:40.394631 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 03:06:40.394683 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":107,"threads_started":1}
I20260924 03:06:40.394776 20555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:40.390224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36069 (local address 127.0.0.1:49690)
0924 03:06:40.390495 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:40.390542 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:40.390629 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:40.391284 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:40.391287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:40.391289 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:40.391378 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:40.391380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.392520 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.392521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:40.393567 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:40.393570 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.394406 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.394407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:40.394506 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:40.394525 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 03:06:40.394604 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 03:06:40.394678 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":10,"thread_start_us":109,"threads_started":1}
I20260924 03:06:40.395519 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:40.390752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57444 (local address 127.0.98.2:33573)
0924 03:06:40.390921 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:40.390926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:40.391026 (+ 100us) server_negotiation.cc:407] Connection header received
0924 03:06:40.391097 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:40.391099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:40.391117 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:40.391196 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:40.391476 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.392810 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.394319 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.395185 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.395257 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:40.395343 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:40.395399 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 03:06:40.395438 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
I20260924 03:06:40.395678 20554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:40.390541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33573 (local address 127.0.0.1:57444)
0924 03:06:40.390945 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:40.390955 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:40.391020 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:40.391284 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:40.391287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:40.391292 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:40.391378 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:40.391381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.392919 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.392920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:40.394216 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:40.394219 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.395258 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.395259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:40.395322 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:40.395327 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:06:40.395382 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 03:06:40.395426 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":3,"thread_start_us":69,"threads_started":1}
I20260924 03:06:40.395761 20480 tablet_service.cc:1467] Processing CreateTablet for tablet 0fb7715f447c4cc49d07245bbab77b7a (DEFAULT_TABLE table=test-table [id=4dd457fb546f46769247818d0ab8f4c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:06:40.396224 20480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fb7715f447c4cc49d07245bbab77b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:40.396405 20349 tablet_service.cc:1467] Processing CreateTablet for tablet 0fb7715f447c4cc49d07245bbab77b7a (DEFAULT_TABLE table=test-table [id=4dd457fb546f46769247818d0ab8f4c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:06:40.396528 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:40.390304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40881 (local address 127.0.0.1:44666)
0924 03:06:40.390684 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:40.390696 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:40.390754 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:40.391764 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:40.391765 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:40.391768 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:40.391816 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:40.391818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.393230 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.393231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:40.394414 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:40.394416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.396290 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.396291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:40.396358 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:40.396363 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:06:40.396415 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 03:06:40.396460 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":125,"threads_started":1}
I20260924 03:06:40.396773 20349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fb7715f447c4cc49d07245bbab77b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:40.396940 20556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:40.390419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44666 (local address 127.0.98.1:40881)
0924 03:06:40.391545 (+ 1126us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:40.391550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:40.391561 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:06:40.391633 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:40.391636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:40.391653 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:40.391736 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:40.391904 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.393111 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.394519 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.396195 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.396281 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:40.396453 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:40.396484 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 03:06:40.396526 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1074,"thread_start_us":86,"threads_started":1}
I20260924 03:06:40.397742 20218 tablet_service.cc:1467] Processing CreateTablet for tablet 0fb7715f447c4cc49d07245bbab77b7a (DEFAULT_TABLE table=test-table [id=4dd457fb546f46769247818d0ab8f4c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:06:40.398113 20218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fb7715f447c4cc49d07245bbab77b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:40.400048 20560 tablet_bootstrap.cc:492] T 0fb7715f447c4cc49d07245bbab77b7a P a7a1d065bb034d61993b9f609a482b51: Bootstrap starting.
I20260924 03:06:40.400303 20561 tablet_bootstrap.cc:492] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de: Bootstrap starting.
I20260924 03:06:40.400954 20560 tablet_bootstrap.cc:654] T 0fb7715f447c4cc49d07245bbab77b7a P a7a1d065bb034d61993b9f609a482b51: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:40.400955 20561 tablet_bootstrap.cc:654] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:40.401309 20560 log.cc:824] T 0fb7715f447c4cc49d07245bbab77b7a P a7a1d065bb034d61993b9f609a482b51: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:40.401309 20561 log.cc:824] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:40.401767 20564 tablet_bootstrap.cc:492] T 0fb7715f447c4cc49d07245bbab77b7a P e554acf2059047feb4f6dd7847a119b8: Bootstrap starting.
I20260924 03:06:40.402212 20560 tablet_bootstrap.cc:492] T 0fb7715f447c4cc49d07245bbab77b7a P a7a1d065bb034d61993b9f609a482b51: No bootstrap required, opened a new log
I20260924 03:06:40.402305 20560 ts_tablet_manager.cc:1397] T 0fb7715f447c4cc49d07245bbab77b7a P a7a1d065bb034d61993b9f609a482b51: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:06:40.402441 20561 tablet_bootstrap.cc:492] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de: No bootstrap required, opened a new log
I20260924 03:06:40.402493 20564 tablet_bootstrap.cc:654] T 0fb7715f447c4cc49d07245bbab77b7a P e554acf2059047feb4f6dd7847a119b8: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:40.402529 20561 ts_tablet_manager.cc:1397] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:06:40.402828 20564 log.cc:824] T 0fb7715f447c4cc49d07245bbab77b7a P e554acf2059047feb4f6dd7847a119b8: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:40.404078 20564 tablet_bootstrap.cc:492] T 0fb7715f447c4cc49d07245bbab77b7a P e554acf2059047feb4f6dd7847a119b8: No bootstrap required, opened a new log
I20260924 03:06:40.404171 20564 ts_tablet_manager.cc:1397] T 0fb7715f447c4cc49d07245bbab77b7a P e554acf2059047feb4f6dd7847a119b8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:06:40.404973 20560 raft_consensus.cc:348] T 0fb7715f447c4cc49d07245bbab77b7a P a7a1d065bb034d61993b9f609a482b51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e554acf2059047feb4f6dd7847a119b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40881 } } peers { permanent_uuid: "a7a1d065bb034d61993b9f609a482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33573 } } peers { permanent_uuid: "017fca85f756452095ef7bef49ccd4de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36069 } }
I20260924 03:06:40.405500 20560 raft_consensus.cc:374] T 0fb7715f447c4cc49d07245bbab77b7a P a7a1d065bb034d61993b9f609a482b51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:40.405685 20560 raft_consensus.cc:729] T 0fb7715f447c4cc49d07245bbab77b7a P a7a1d065bb034d61993b9f609a482b51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7a1d065bb034d61993b9f609a482b51, State: Initialized, Role: FOLLOWER
I20260924 03:06:40.405941 20560 consensus_queue.cc:260] T 0fb7715f447c4cc49d07245bbab77b7a P a7a1d065bb034d61993b9f609a482b51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e554acf2059047feb4f6dd7847a119b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40881 } } peers { permanent_uuid: "a7a1d065bb034d61993b9f609a482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33573 } } peers { permanent_uuid: "017fca85f756452095ef7bef49ccd4de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36069 } }
I20260924 03:06:40.406296 20560 ts_tablet_manager.cc:1428] T 0fb7715f447c4cc49d07245bbab77b7a P a7a1d065bb034d61993b9f609a482b51: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 03:06:40.406459 20415 heartbeater.cc:503] Master 127.0.98.62:44257 was elected leader, sending a full tablet report...
I20260924 03:06:40.406709 20561 raft_consensus.cc:348] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e554acf2059047feb4f6dd7847a119b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40881 } } peers { permanent_uuid: "a7a1d065bb034d61993b9f609a482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33573 } } peers { permanent_uuid: "017fca85f756452095ef7bef49ccd4de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36069 } }
I20260924 03:06:40.406944 20561 raft_consensus.cc:374] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:40.407001 20561 raft_consensus.cc:729] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017fca85f756452095ef7bef49ccd4de, State: Initialized, Role: FOLLOWER
I20260924 03:06:40.407121 20561 consensus_queue.cc:260] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e554acf2059047feb4f6dd7847a119b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40881 } } peers { permanent_uuid: "a7a1d065bb034d61993b9f609a482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33573 } } peers { permanent_uuid: "017fca85f756452095ef7bef49ccd4de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36069 } }
I20260924 03:06:40.407383 20561 ts_tablet_manager.cc:1428] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260924 03:06:40.407568 20546 heartbeater.cc:503] Master 127.0.98.62:44257 was elected leader, sending a full tablet report...
I20260924 03:06:40.407855 20564 raft_consensus.cc:348] T 0fb7715f447c4cc49d07245bbab77b7a P e554acf2059047feb4f6dd7847a119b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e554acf2059047feb4f6dd7847a119b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40881 } } peers { permanent_uuid: "a7a1d065bb034d61993b9f609a482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33573 } } peers { permanent_uuid: "017fca85f756452095ef7bef49ccd4de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36069 } }
I20260924 03:06:40.408027 20564 raft_consensus.cc:374] T 0fb7715f447c4cc49d07245bbab77b7a P e554acf2059047feb4f6dd7847a119b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:40.408067 20564 raft_consensus.cc:729] T 0fb7715f447c4cc49d07245bbab77b7a P e554acf2059047feb4f6dd7847a119b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e554acf2059047feb4f6dd7847a119b8, State: Initialized, Role: FOLLOWER
I20260924 03:06:40.408217 20564 consensus_queue.cc:260] T 0fb7715f447c4cc49d07245bbab77b7a P e554acf2059047feb4f6dd7847a119b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e554acf2059047feb4f6dd7847a119b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40881 } } peers { permanent_uuid: "a7a1d065bb034d61993b9f609a482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33573 } } peers { permanent_uuid: "017fca85f756452095ef7bef49ccd4de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36069 } }
I20260924 03:06:40.408517 20564 ts_tablet_manager.cc:1428] T 0fb7715f447c4cc49d07245bbab77b7a P e554acf2059047feb4f6dd7847a119b8: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260924 03:06:40.408617 20284 heartbeater.cc:503] Master 127.0.98.62:44257 was elected leader, sending a full tablet report...
I20260924 03:06:40.442340 20567 raft_consensus.cc:482] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:06:40.442550 20567 raft_consensus.cc:504] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e554acf2059047feb4f6dd7847a119b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40881 } } peers { permanent_uuid: "a7a1d065bb034d61993b9f609a482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33573 } } peers { permanent_uuid: "017fca85f756452095ef7bef49ccd4de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36069 } }
I20260924 03:06:40.443451 20567 leader_election.cc:288] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e554acf2059047feb4f6dd7847a119b8 (127.0.98.1:40881), a7a1d065bb034d61993b9f609a482b51 (127.0.98.2:33573)
I20260924 03:06:40.447494 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:40.443673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40881 (local address 127.0.98.3:44751)
0924 03:06:40.443727 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:40.443741 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:40.443805 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:40.444019 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:40.444020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:40.444022 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:40.444105 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:40.444108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.445001 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.445002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:40.445923 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:40.445927 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.447053 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.447055 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:40.447151 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:40.447156 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:06:40.447347 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 03:06:40.447401 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 03:06:40.447500 20556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:40.443765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44751 (local address 127.0.98.1:40881)
0924 03:06:40.443830 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:40.443832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:40.443842 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:06:40.443873 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:40.443874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:40.443887 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:40.443948 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:40.444223 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.444895 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.446088 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.446921 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.447095 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:40.447120 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:40.447398 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 03:06:40.447444 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 03:06:40.447952 20238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb7715f447c4cc49d07245bbab77b7a" candidate_uuid: "017fca85f756452095ef7bef49ccd4de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e554acf2059047feb4f6dd7847a119b8" is_pre_election: true
I20260924 03:06:40.448139 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:40.443887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57939 (local address 127.0.98.2:33573)
0924 03:06:40.443941 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:40.443943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:40.444099 (+ 156us) server_negotiation.cc:407] Connection header received
0924 03:06:40.444149 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:40.444150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:40.444166 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:40.444232 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:06:40.444374 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.445085 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.447050 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.447736 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.447814 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:40.447826 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:40.448024 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 03:06:40.448073 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 03:06:40.448158 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:40.443822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33573 (local address 127.0.98.3:57939)
0924 03:06:40.444026 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:40.444035 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:06:40.444088 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:06:40.444247 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:06:40.444255 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:06:40.444257 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:06:40.444322 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:40.444324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.445285 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.445287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:40.446918 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:06:40.446921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.447838 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.447839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:06:40.447922 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:40.447926 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:06:40.448012 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 03:06:40.448057 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":130,"threads_started":1}
I20260924 03:06:40.448174 20238 raft_consensus.cc:2454] T 0fb7715f447c4cc49d07245bbab77b7a P e554acf2059047feb4f6dd7847a119b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 017fca85f756452095ef7bef49ccd4de in term 0.
I20260924 03:06:40.448493 20369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb7715f447c4cc49d07245bbab77b7a" candidate_uuid: "017fca85f756452095ef7bef49ccd4de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7a1d065bb034d61993b9f609a482b51" is_pre_election: true
I20260924 03:06:40.448633 20369 raft_consensus.cc:2454] T 0fb7715f447c4cc49d07245bbab77b7a P a7a1d065bb034d61993b9f609a482b51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 017fca85f756452095ef7bef49ccd4de in term 0.
I20260924 03:06:40.448858 20435 leader_election.cc:302] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017fca85f756452095ef7bef49ccd4de, e554acf2059047feb4f6dd7847a119b8; no voters:
I20260924 03:06:40.449049 20567 raft_consensus.cc:2781] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:06:40.449131 20567 raft_consensus.cc:482] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:06:40.449159 20567 raft_consensus.cc:3037] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:40.449934 20567 raft_consensus.cc:504] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e554acf2059047feb4f6dd7847a119b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40881 } } peers { permanent_uuid: "a7a1d065bb034d61993b9f609a482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33573 } } peers { permanent_uuid: "017fca85f756452095ef7bef49ccd4de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36069 } }
I20260924 03:06:40.450094 20567 leader_election.cc:288] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de [CANDIDATE]: Term 1 election: Requested vote from peers e554acf2059047feb4f6dd7847a119b8 (127.0.98.1:40881), a7a1d065bb034d61993b9f609a482b51 (127.0.98.2:33573)
I20260924 03:06:40.450489 20238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb7715f447c4cc49d07245bbab77b7a" candidate_uuid: "017fca85f756452095ef7bef49ccd4de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e554acf2059047feb4f6dd7847a119b8"
I20260924 03:06:40.450503 20369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb7715f447c4cc49d07245bbab77b7a" candidate_uuid: "017fca85f756452095ef7bef49ccd4de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7a1d065bb034d61993b9f609a482b51"
I20260924 03:06:40.450596 20369 raft_consensus.cc:3037] T 0fb7715f447c4cc49d07245bbab77b7a P a7a1d065bb034d61993b9f609a482b51 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:40.450596 20238 raft_consensus.cc:3037] T 0fb7715f447c4cc49d07245bbab77b7a P e554acf2059047feb4f6dd7847a119b8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:40.451584 20369 raft_consensus.cc:2454] T 0fb7715f447c4cc49d07245bbab77b7a P a7a1d065bb034d61993b9f609a482b51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017fca85f756452095ef7bef49ccd4de in term 1.
I20260924 03:06:40.451584 20238 raft_consensus.cc:2454] T 0fb7715f447c4cc49d07245bbab77b7a P e554acf2059047feb4f6dd7847a119b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017fca85f756452095ef7bef49ccd4de in term 1.
I20260924 03:06:40.451833 20435 leader_election.cc:302] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017fca85f756452095ef7bef49ccd4de, a7a1d065bb034d61993b9f609a482b51; no voters:
I20260924 03:06:40.451956 20567 raft_consensus.cc:2781] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:06:40.452185 20567 raft_consensus.cc:686] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de [term 1 LEADER]: Becoming Leader. State: Replica: 017fca85f756452095ef7bef49ccd4de, State: Running, Role: LEADER
I20260924 03:06:40.452306 20567 consensus_queue.cc:237] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e554acf2059047feb4f6dd7847a119b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40881 } } peers { permanent_uuid: "a7a1d065bb034d61993b9f609a482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33573 } } peers { permanent_uuid: "017fca85f756452095ef7bef49ccd4de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36069 } }
I20260924 03:06:40.453286 20096 catalog_manager.cc:5495] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de reported cstate change: term changed from 0 to 1, leader changed from <none> to 017fca85f756452095ef7bef49ccd4de (127.0.98.3). New cstate: current_term: 1 leader_uuid: "017fca85f756452095ef7bef49ccd4de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e554acf2059047feb4f6dd7847a119b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7a1d065bb034d61993b9f609a482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017fca85f756452095ef7bef49ccd4de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36069 } health_report { overall_health: HEALTHY } } }
I20260924 03:06:40.460453 20557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:40.457584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49700 (local address 127.0.98.3:36069)
0924 03:06:40.457669 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:40.457671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:40.457681 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:06:40.457717 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:40.457718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:40.457735 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:40.457821 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:06:40.458032 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.458640 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.459712 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:40.459901 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:40.459966 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:40.460096 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:06:40.460205 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:06:40.460269 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:40.460347 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 03:06:40.460372 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 03:06:40.462662 20369 raft_consensus.cc:1261] T 0fb7715f447c4cc49d07245bbab77b7a P a7a1d065bb034d61993b9f609a482b51 [term 1 FOLLOWER]: Refusing update from remote peer 017fca85f756452095ef7bef49ccd4de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:06:40.462671 20238 raft_consensus.cc:1261] T 0fb7715f447c4cc49d07245bbab77b7a P e554acf2059047feb4f6dd7847a119b8 [term 1 FOLLOWER]: Refusing update from remote peer 017fca85f756452095ef7bef49ccd4de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:06:40.463222 20567 consensus_queue.cc:1035] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de [LEADER]: Connected to new peer: Peer: permanent_uuid: "e554acf2059047feb4f6dd7847a119b8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:06:40.463397 20570 consensus_queue.cc:1035] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7a1d065bb034d61993b9f609a482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:06:40.467257 20574 mvcc.cc:205] Tried to move back new op lower bound from 7332737845092040704 to 7332737845053616128. Current Snapshot: MvccSnapshot[applied={T|T < 7332737845092040704}]
I20260924 03:06:40.467571 20575 mvcc.cc:205] Tried to move back new op lower bound from 7332737845092040704 to 7332737845053616128. Current Snapshot: MvccSnapshot[applied={T|T < 7332737845092040704}]
I20260924 03:06:40.470638 20096 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:06:40.470765 20096 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:06:40.471758 20096 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 991258a83f434e4d8fb536239bbead4c: Sending DeleteTablet for 3 replicas of tablet 0fb7715f447c4cc49d07245bbab77b7a
I20260924 03:06:40.472126 20218 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fb7715f447c4cc49d07245bbab77b7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:06:40 UTC) from {username='slave'} at 127.0.0.1:44666
I20260924 03:06:40.472165 20349 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fb7715f447c4cc49d07245bbab77b7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:06:40 UTC) from {username='slave'} at 127.0.0.1:57444
I20260924 03:06:40.472159 20480 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fb7715f447c4cc49d07245bbab77b7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:06:40 UTC) from {username='slave'} at 127.0.0.1:49690
I20260924 03:06:40.472438 20585 tablet_replica.cc:330] T 0fb7715f447c4cc49d07245bbab77b7a P a7a1d065bb034d61993b9f609a482b51: stopping tablet replica
I20260924 03:06:40.472558 20585 raft_consensus.cc:2229] T 0fb7715f447c4cc49d07245bbab77b7a P a7a1d065bb034d61993b9f609a482b51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:06:40.472719 20585 raft_consensus.cc:2258] T 0fb7715f447c4cc49d07245bbab77b7a P a7a1d065bb034d61993b9f609a482b51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:06:40.472720 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 20156
I20260924 03:06:40.473201 20585 ts_tablet_manager.cc:1905] T 0fb7715f447c4cc49d07245bbab77b7a P a7a1d065bb034d61993b9f609a482b51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:06:40.475311 20585 ts_tablet_manager.cc:1918] T 0fb7715f447c4cc49d07245bbab77b7a P a7a1d065bb034d61993b9f609a482b51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:06:40.475402 20585 log.cc:1196] T 0fb7715f447c4cc49d07245bbab77b7a P a7a1d065bb034d61993b9f609a482b51: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-1/wal/wals/0fb7715f447c4cc49d07245bbab77b7a
I20260924 03:06:40.475706 20585 ts_tablet_manager.cc:1939] T 0fb7715f447c4cc49d07245bbab77b7a P a7a1d065bb034d61993b9f609a482b51: Deleting consensus metadata
I20260924 03:06:40.475840 20584 tablet_replica.cc:330] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de: stopping tablet replica
I20260924 03:06:40.476022 20584 raft_consensus.cc:2229] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:06:40.476221 20584 raft_consensus.cc:2258] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:06:40.476234 20082 catalog_manager.cc:4841] TS a7a1d065bb034d61993b9f609a482b51 (127.0.98.2:33573): tablet 0fb7715f447c4cc49d07245bbab77b7a (table test-table [id=4dd457fb546f46769247818d0ab8f4c1]) successfully deleted
I20260924 03:06:40.476807 20584 ts_tablet_manager.cc:1905] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:06:40.479874 20584 ts_tablet_manager.cc:1918] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:06:40.480113 20584 log.cc:1196] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de: Deleting WAL directory at /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218948286999-392-0/minicluster-data/ts-2/wal/wals/0fb7715f447c4cc49d07245bbab77b7a
I20260924 03:06:40.480520 20584 ts_tablet_manager.cc:1939] T 0fb7715f447c4cc49d07245bbab77b7a P 017fca85f756452095ef7bef49ccd4de: Deleting consensus metadata
I20260924 03:06:40.481205 20083 catalog_manager.cc:4841] TS 017fca85f756452095ef7bef49ccd4de (127.0.98.3:36069): tablet 0fb7715f447c4cc49d07245bbab77b7a (table test-table [id=4dd457fb546f46769247818d0ab8f4c1]) successfully deleted
W20260924 03:06:40.482024 20082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40881 (error 108)
I20260924 03:06:40.482285 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 20287
I20260924 03:06:40.483011 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:40.482457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40881 (local address 127.0.0.1:44682)
0924 03:06:40.482914 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 03:06:40.482962 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431}
W20260924 03:06:40.484047 20082 catalog_manager.cc:4584] TS e554acf2059047feb4f6dd7847a119b8 (127.0.98.1:40881): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0fb7715f447c4cc49d07245bbab77b7a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40881: connect: Connection refused (error 111)
I20260924 03:06:40.489269 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 20418
I20260924 03:06:40.494926 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 20065
2026-09-24T03:06:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (733 ms)
[----------] 6 tests from EncryptionPolicyTest (9269 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:06:40 dist-test-slave-wgsb krb5kdc[20589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:06:40 dist-test-slave-wgsb krb5kdc[20589](info): set up 2 sockets
Sep 24 03:06:40 dist-test-slave-wgsb krb5kdc[20589](info): commencing operation
krb5kdc: starting...
W20260924 03:06:41.583923 392 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:06:41 dist-test-slave-wgsb krb5kdc[20589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:06:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:06:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:06:41.716110 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38583
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:06:41.802058 20602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:41.802320 20602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:41.802359 20602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:41.803578 20602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:06:41.803673 20602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:41.803700 20602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:41.803732 20602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:06:41.803756 20602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:06:41.805259 20602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38583
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:41.805547 20602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:41.805799 20602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:41.811465 20608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:41.811512 20602 server_base.cc:1034] running on GCE node
W20260924 03:06:41.811527 20607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:41.811489 20610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:41.812021 20602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:41.812326 20602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:41.813512 20602 hybrid_clock.cc:648] HybridClock initialized: now 1790219201813489 us; error 58 us; skew 500 ppm
Sep 24 03:06:41 dist-test-slave-wgsb krb5kdc[20589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:41.823333 20602 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:06:41.823886 20602 webserver.cc:460] Webserver started at http://127.0.0.1:34117/ using document root <none> and password file <none>
I20260924 03:06:41.824134 20602 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:41.824206 20602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:41.824358 20602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:41.825489 20602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "8fbc6e963bbd413da3a84a26f66cb45d"
format_stamp: "Formatted at 2026-09-24 03:06:41 on dist-test-slave-wgsb"
server_key: "4a34fee7295b4ec1a10f98327f4fcb9c"
server_key_iv: "0231f0c5cca18ebef1463131e832bcf6"
server_key_version: "encryptionkey@0"
I20260924 03:06:41.825884 20602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "8fbc6e963bbd413da3a84a26f66cb45d"
format_stamp: "Formatted at 2026-09-24 03:06:41 on dist-test-slave-wgsb"
server_key: "4a34fee7295b4ec1a10f98327f4fcb9c"
server_key_iv: "0231f0c5cca18ebef1463131e832bcf6"
server_key_version: "encryptionkey@0"
I20260924 03:06:41.827250 20602 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:06:41.828168 20617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:41.828397 20602 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 03:06:41.828511 20602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "8fbc6e963bbd413da3a84a26f66cb45d"
format_stamp: "Formatted at 2026-09-24 03:06:41 on dist-test-slave-wgsb"
server_key: "4a34fee7295b4ec1a10f98327f4fcb9c"
server_key_iv: "0231f0c5cca18ebef1463131e832bcf6"
server_key_version: "encryptionkey@0"
I20260924 03:06:41.828603 20602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:41.839957 20602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:41.842106 20602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:41.842258 20602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:41.846391 20602 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38583
I20260924 03:06:41.846429 20669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38583 every 8 connection(s)
I20260924 03:06:41.847021 20602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:06:41.847795 20670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:41.850050 20670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fbc6e963bbd413da3a84a26f66cb45d: Bootstrap starting.
I20260924 03:06:41.850641 20670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fbc6e963bbd413da3a84a26f66cb45d: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:41.850926 20670 log.cc:824] T 00000000000000000000000000000000 P 8fbc6e963bbd413da3a84a26f66cb45d: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:41.851697 20670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fbc6e963bbd413da3a84a26f66cb45d: No bootstrap required, opened a new log
I20260924 03:06:41.851953 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 20602
I20260924 03:06:41.852090 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:06:41.852283 392 external_mini_cluster.cc:1428] Setting key 601ed4cd037164eb8b25b2185565e1b6
I20260924 03:06:41.853642 20670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fbc6e963bbd413da3a84a26f66cb45d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fbc6e963bbd413da3a84a26f66cb45d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38583 } }
I20260924 03:06:41.853812 20670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fbc6e963bbd413da3a84a26f66cb45d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:41.853858 20670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fbc6e963bbd413da3a84a26f66cb45d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fbc6e963bbd413da3a84a26f66cb45d, State: Initialized, Role: FOLLOWER
I20260924 03:06:41.853993 20670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fbc6e963bbd413da3a84a26f66cb45d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fbc6e963bbd413da3a84a26f66cb45d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38583 } }
I20260924 03:06:41.854080 20670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fbc6e963bbd413da3a84a26f66cb45d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:06:41.854125 20670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fbc6e963bbd413da3a84a26f66cb45d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:06:41.854204 20670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fbc6e963bbd413da3a84a26f66cb45d [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:41.855227 20670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fbc6e963bbd413da3a84a26f66cb45d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fbc6e963bbd413da3a84a26f66cb45d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38583 } }
I20260924 03:06:41.855355 20670 leader_election.cc:302] T 00000000000000000000000000000000 P 8fbc6e963bbd413da3a84a26f66cb45d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fbc6e963bbd413da3a84a26f66cb45d; no voters:
I20260924 03:06:41.855556 20670 leader_election.cc:288] T 00000000000000000000000000000000 P 8fbc6e963bbd413da3a84a26f66cb45d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:06:41.855579 20675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fbc6e963bbd413da3a84a26f66cb45d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:06:41.855726 20675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fbc6e963bbd413da3a84a26f66cb45d [term 1 LEADER]: Becoming Leader. State: Replica: 8fbc6e963bbd413da3a84a26f66cb45d, State: Running, Role: LEADER
I20260924 03:06:41.855809 20675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fbc6e963bbd413da3a84a26f66cb45d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fbc6e963bbd413da3a84a26f66cb45d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38583 } }
I20260924 03:06:41.855875 20670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fbc6e963bbd413da3a84a26f66cb45d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:06:41.856132 20675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fbc6e963bbd413da3a84a26f66cb45d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fbc6e963bbd413da3a84a26f66cb45d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fbc6e963bbd413da3a84a26f66cb45d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38583 } } }
I20260924 03:06:41.856204 20675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fbc6e963bbd413da3a84a26f66cb45d [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:41.856340 20675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fbc6e963bbd413da3a84a26f66cb45d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fbc6e963bbd413da3a84a26f66cb45d. Latest consensus state: current_term: 1 leader_uuid: "8fbc6e963bbd413da3a84a26f66cb45d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fbc6e963bbd413da3a84a26f66cb45d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38583 } } }
I20260924 03:06:41.856410 20675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fbc6e963bbd413da3a84a26f66cb45d [sys.catalog]: This master's current role is: LEADER
W20260924 03:06:41.857772 20691 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8fbc6e963bbd413da3a84a26f66cb45d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:06:41.857880 20691 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:06:41.857954 20679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:06:41.858147 20679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 03:06:41 dist-test-slave-wgsb krb5kdc[20589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 03:06:41.860311 20679 catalog_manager.cc:1347] Generated new cluster ID: aa16ca0ecc7944dca05d556d143825a9
I20260924 03:06:41.860363 20679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:06:41.864810 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:41.852871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37893 (local address 127.0.0.1:38583)
0924 03:06:41.853056 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:41.853063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:41.853087 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:06:41.853621 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:41.853628 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:41.853673 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:41.853823 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:41.854601 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:41.855654 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:41.856708 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:41.857401 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:41.857510 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:41.860934 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:41.860938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:41.860939 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:06:41.860951 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:41.862900 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:41.863387 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:41.863390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:41.863391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:41.863511 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:41.863785 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:41.863786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:41.863786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:41.864080 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:41.864427 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:41.864506 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 03:06:41.864604 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":74,"threads_started":1}
I20260924 03:06:41.866050 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:06:41.868013 20679 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:06:41.868709 20679 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:06:41.883579 20679 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fbc6e963bbd413da3a84a26f66cb45d: Generated new TSK 0
I20260924 03:06:41.883914 20679 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:06:41.889961 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:41.867866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53867 (local address 127.0.0.1:38583)
0924 03:06:41.867899 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:41.867901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:41.876796 (+ 8895us) server_negotiation.cc:407] Connection header received
0924 03:06:41.876882 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:41.876884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:41.876913 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:41.877104 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:41.878096 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:41.878676 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:41.884368 (+ 5692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:41.884977 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:41.885047 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:41.887136 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:41.887139 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:41.887139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:06:41.887149 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:41.888679 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:41.889120 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:41.889122 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:41.889122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:41.889184 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:41.889414 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:41.889415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:41.889415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:41.889542 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:41.889658 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:41.889777 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 03:06:41.889843 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 03:06:41.892349 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 20602
W20260924 03:06:41.896286 20587 connection.cc:535] client connection to 127.0.0.1:38583 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38583): Connection reset by peer (error 104)
2026-09-24T03:06:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1392 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:06:41 dist-test-slave-wgsb krb5kdc[20701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:06:41 dist-test-slave-wgsb krb5kdc[20701](info): set up 2 sockets
Sep 24 03:06:41 dist-test-slave-wgsb krb5kdc[20701](info): commencing operation
krb5kdc: starting...
W20260924 03:06:42.971748 392 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:06:43 dist-test-slave-wgsb krb5kdc[20701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:06:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:06:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:06:43.099445 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35299
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:06:43.183365 20714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:43.183646 20714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:43.183699 20714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:43.184845 20714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:06:43.184914 20714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:43.184940 20714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:43.184970 20714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:06:43.185003 20714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:06:43.186540 20714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35299
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:43.186820 20714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:43.187047 20714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:43.192209 20720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:43.192227 20719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:43.192220 20722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:43.192596 20714 server_base.cc:1034] running on GCE node
I20260924 03:06:43.192834 20714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:43.193131 20714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:43.194368 20714 hybrid_clock.cc:648] HybridClock initialized: now 1790219203194356 us; error 28 us; skew 500 ppm
Sep 24 03:06:43 dist-test-slave-wgsb krb5kdc[20701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:43.197335 20714 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:06:43.197791 20714 webserver.cc:460] Webserver started at http://127.0.0.1:38385/ using document root <none> and password file <none>
I20260924 03:06:43.198035 20714 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:43.198081 20714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:43.198287 20714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:43.199384 20714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "1eaa010b08c343a099312d412af806fa"
format_stamp: "Formatted at 2026-09-24 03:06:43 on dist-test-slave-wgsb"
server_key: "94d5ae464499908676ec6906af11e3da"
server_key_iv: "fff894ccf74e94369578fd186c55976f"
server_key_version: "encryptionkey@0"
I20260924 03:06:43.199770 20714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "1eaa010b08c343a099312d412af806fa"
format_stamp: "Formatted at 2026-09-24 03:06:43 on dist-test-slave-wgsb"
server_key: "94d5ae464499908676ec6906af11e3da"
server_key_iv: "fff894ccf74e94369578fd186c55976f"
server_key_version: "encryptionkey@0"
I20260924 03:06:43.201138 20714 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:06:43.201982 20729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:43.202170 20714 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:06:43.202399 20714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "1eaa010b08c343a099312d412af806fa"
format_stamp: "Formatted at 2026-09-24 03:06:43 on dist-test-slave-wgsb"
server_key: "94d5ae464499908676ec6906af11e3da"
server_key_iv: "fff894ccf74e94369578fd186c55976f"
server_key_version: "encryptionkey@0"
I20260924 03:06:43.202462 20714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:43.219856 20714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:43.222265 20714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:43.222402 20714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:43.226686 20714 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35299
I20260924 03:06:43.226719 20781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35299 every 8 connection(s)
I20260924 03:06:43.227118 20714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:06:43.228144 20782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:43.231385 20782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eaa010b08c343a099312d412af806fa: Bootstrap starting.
I20260924 03:06:43.232064 20782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1eaa010b08c343a099312d412af806fa: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:43.232373 20782 log.cc:824] T 00000000000000000000000000000000 P 1eaa010b08c343a099312d412af806fa: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:43.233103 20782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eaa010b08c343a099312d412af806fa: No bootstrap required, opened a new log
I20260924 03:06:43.234611 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 20714
I20260924 03:06:43.234751 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:06:43.234948 392 external_mini_cluster.cc:1428] Setting key beff846c6eb3baac5cc6432c853bc9f0
I20260924 03:06:43.235028 20782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1eaa010b08c343a099312d412af806fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eaa010b08c343a099312d412af806fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35299 } }
I20260924 03:06:43.235195 20782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1eaa010b08c343a099312d412af806fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:43.235224 20782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1eaa010b08c343a099312d412af806fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eaa010b08c343a099312d412af806fa, State: Initialized, Role: FOLLOWER
I20260924 03:06:43.235409 20782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1eaa010b08c343a099312d412af806fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eaa010b08c343a099312d412af806fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35299 } }
I20260924 03:06:43.235499 20782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1eaa010b08c343a099312d412af806fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:06:43.235533 20782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1eaa010b08c343a099312d412af806fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:06:43.235584 20782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1eaa010b08c343a099312d412af806fa [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:43.236441 20782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1eaa010b08c343a099312d412af806fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eaa010b08c343a099312d412af806fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35299 } }
I20260924 03:06:43.236577 20782 leader_election.cc:302] T 00000000000000000000000000000000 P 1eaa010b08c343a099312d412af806fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1eaa010b08c343a099312d412af806fa; no voters:
I20260924 03:06:43.236785 20782 leader_election.cc:288] T 00000000000000000000000000000000 P 1eaa010b08c343a099312d412af806fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:06:43.236816 20787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1eaa010b08c343a099312d412af806fa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:06:43.236931 20787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1eaa010b08c343a099312d412af806fa [term 1 LEADER]: Becoming Leader. State: Replica: 1eaa010b08c343a099312d412af806fa, State: Running, Role: LEADER
I20260924 03:06:43.237015 20787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1eaa010b08c343a099312d412af806fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eaa010b08c343a099312d412af806fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35299 } }
I20260924 03:06:43.237061 20782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1eaa010b08c343a099312d412af806fa [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:06:43.237408 20788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eaa010b08c343a099312d412af806fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1eaa010b08c343a099312d412af806fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eaa010b08c343a099312d412af806fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35299 } } }
I20260924 03:06:43.237588 20788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eaa010b08c343a099312d412af806fa [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:43.237393 20787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eaa010b08c343a099312d412af806fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1eaa010b08c343a099312d412af806fa. Latest consensus state: current_term: 1 leader_uuid: "1eaa010b08c343a099312d412af806fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eaa010b08c343a099312d412af806fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35299 } } }
I20260924 03:06:43.237727 20787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eaa010b08c343a099312d412af806fa [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:43.237974 20794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:06:43.238724 20794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:06:43.240856 20794 catalog_manager.cc:1347] Generated new cluster ID: 3d3097c69cc54e28939093c538dc4be9
I20260924 03:06:43.240959 20794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:06:43.241662 20786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:43.235498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50275 (local address 127.0.0.1:35299)
0924 03:06:43.235945 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:43.235953 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:43.235983 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:06:43.236090 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:43.236098 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:43.236154 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:43.236302 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:43.236531 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:43.237958 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:43.239271 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:43.239900 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:43.240003 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:43.240821 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:43.240826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:43.240826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:43.241102 (+ 276us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:43.241146 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:43.241163 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:43.241279 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:43.241400 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 03:06:43.241434 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":123,"threads_started":1}
I20260924 03:06:43.242719 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:06:43.248546 20786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:43.244486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36609 (local address 127.0.0.1:35299)
0924 03:06:43.244542 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:43.244544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:43.244743 (+ 199us) server_negotiation.cc:407] Connection header received
0924 03:06:43.244821 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:43.244822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:43.244843 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:43.244897 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:43.245073 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:43.245800 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:43.246929 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:43.247256 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:43.247311 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:43.248222 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:43.248224 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:43.248224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:43.248229 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:43.248245 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:43.248249 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:43.248305 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:43.248448 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 03:06:43.248463 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 03:06:43.250977 20794 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:06:43.251821 20794 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:06:43.257252 20794 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1eaa010b08c343a099312d412af806fa: Generated new TSK 0
I20260924 03:06:43.257473 20794 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:06:43.260910 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 20714
W20260924 03:06:43.265954 20699 connection.cc:535] client connection to 127.0.0.1:35299 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35299): Connection reset by peer (error 104)
2026-09-24T03:06:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:06:43 dist-test-slave-wgsb krb5kdc[20813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:06:43 dist-test-slave-wgsb krb5kdc[20813](info): set up 2 sockets
Sep 24 03:06:43 dist-test-slave-wgsb krb5kdc[20813](info): commencing operation
krb5kdc: starting...
W20260924 03:06:44.331815 392 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:06:44 dist-test-slave-wgsb krb5kdc[20813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:06:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:06:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:06:44.459911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38159
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:06:44.544750 20826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:44.544952 20826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:44.544975 20826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:44.546130 20826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:06:44.546171 20826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:44.546186 20826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:44.546200 20826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:06:44.546212 20826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:06:44.547582 20826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38159
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:44.547838 20826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:44.547998 20826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:06:44.552871 20826 server_base.cc:1034] running on GCE node
W20260924 03:06:44.552994 20831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:44.552999 20834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:44.552902 20832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:44.553347 20826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:44.553586 20826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:44.554742 20826 hybrid_clock.cc:648] HybridClock initialized: now 1790219204554720 us; error 36 us; skew 500 ppm
Sep 24 03:06:44 dist-test-slave-wgsb krb5kdc[20813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:44.557448 20826 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:06:44.557893 20826 webserver.cc:460] Webserver started at http://127.0.0.1:39753/ using document root <none> and password file <none>
I20260924 03:06:44.558128 20826 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:44.558174 20826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:44.558341 20826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:44.559453 20826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "e55de148e11f448b966bc7f729d6c3a6"
format_stamp: "Formatted at 2026-09-24 03:06:44 on dist-test-slave-wgsb"
server_key: "2ef40441ee3770e506628443b5f5fef7"
server_key_iv: "612e9517bc7ab59e27d72844ea00e3dc"
server_key_version: "encryptionkey@0"
I20260924 03:06:44.559890 20826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "e55de148e11f448b966bc7f729d6c3a6"
format_stamp: "Formatted at 2026-09-24 03:06:44 on dist-test-slave-wgsb"
server_key: "2ef40441ee3770e506628443b5f5fef7"
server_key_iv: "612e9517bc7ab59e27d72844ea00e3dc"
server_key_version: "encryptionkey@0"
I20260924 03:06:44.561228 20826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:06:44.562292 20841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:44.562707 20826 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:06:44.562821 20826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "e55de148e11f448b966bc7f729d6c3a6"
format_stamp: "Formatted at 2026-09-24 03:06:44 on dist-test-slave-wgsb"
server_key: "2ef40441ee3770e506628443b5f5fef7"
server_key_iv: "612e9517bc7ab59e27d72844ea00e3dc"
server_key_version: "encryptionkey@0"
I20260924 03:06:44.562920 20826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:44.567866 20826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:44.572670 20826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:44.573130 20826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:44.585736 20826 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38159
I20260924 03:06:44.585757 20893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38159 every 8 connection(s)
I20260924 03:06:44.586222 20826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:06:44.586699 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 20826
I20260924 03:06:44.586854 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:06:44.587432 392 external_mini_cluster.cc:1428] Setting key 04de2e6bc41d5acf2c48ae699fdfd4dd
I20260924 03:06:44.588301 20894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:44.592906 20897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:44.588447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50851 (local address 127.0.0.1:38159)
0924 03:06:44.588687 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:44.588697 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:44.588732 (+ 35us) server_negotiation.cc:407] Connection header received
0924 03:06:44.589533 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:44.589550 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:44.589609 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:44.589892 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:44.592097 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:44.592105 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:44.592105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:44.592114 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:44.592209 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:44.592231 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:44.592359 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:44.592457 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 03:06:44.592536 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":94,"threads_started":1}
I20260924 03:06:44.593779 20894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e55de148e11f448b966bc7f729d6c3a6: Bootstrap starting.
I20260924 03:06:44.594856 20894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e55de148e11f448b966bc7f729d6c3a6: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:44.596004 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:06:44.595990 20894 log.cc:824] T 00000000000000000000000000000000 P e55de148e11f448b966bc7f729d6c3a6: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:44.597546 20894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e55de148e11f448b966bc7f729d6c3a6: No bootstrap required, opened a new log
I20260924 03:06:44.600286 20897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:44.597558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44601 (local address 127.0.0.1:38159)
0924 03:06:44.597807 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:44.597810 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:44.597823 (+ 13us) server_negotiation.cc:407] Connection header received
0924 03:06:44.597850 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:44.597851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:44.597857 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:44.597944 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:44.599485 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:44.599487 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:44.599488 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:44.599494 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:44.599515 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:44.599520 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:44.599639 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:44.600152 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 03:06:44.600180 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219}
I20260924 03:06:44.601950 20894 raft_consensus.cc:348] T 00000000000000000000000000000000 P e55de148e11f448b966bc7f729d6c3a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e55de148e11f448b966bc7f729d6c3a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38159 } }
I20260924 03:06:44.602672 20894 raft_consensus.cc:374] T 00000000000000000000000000000000 P e55de148e11f448b966bc7f729d6c3a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:44.602805 20894 raft_consensus.cc:729] T 00000000000000000000000000000000 P e55de148e11f448b966bc7f729d6c3a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e55de148e11f448b966bc7f729d6c3a6, State: Initialized, Role: FOLLOWER
I20260924 03:06:44.603500 20894 consensus_queue.cc:260] T 00000000000000000000000000000000 P e55de148e11f448b966bc7f729d6c3a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e55de148e11f448b966bc7f729d6c3a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38159 } }
I20260924 03:06:44.603657 20894 raft_consensus.cc:388] T 00000000000000000000000000000000 P e55de148e11f448b966bc7f729d6c3a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:06:44.603729 20894 raft_consensus.cc:482] T 00000000000000000000000000000000 P e55de148e11f448b966bc7f729d6c3a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:06:44.603797 20894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e55de148e11f448b966bc7f729d6c3a6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:44.604576 20894 raft_consensus.cc:504] T 00000000000000000000000000000000 P e55de148e11f448b966bc7f729d6c3a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e55de148e11f448b966bc7f729d6c3a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38159 } }
I20260924 03:06:44.604776 20894 leader_election.cc:302] T 00000000000000000000000000000000 P e55de148e11f448b966bc7f729d6c3a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e55de148e11f448b966bc7f729d6c3a6; no voters:
I20260924 03:06:44.605088 20894 leader_election.cc:288] T 00000000000000000000000000000000 P e55de148e11f448b966bc7f729d6c3a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:06:44.605226 20904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e55de148e11f448b966bc7f729d6c3a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:06:44.605484 20904 raft_consensus.cc:686] T 00000000000000000000000000000000 P e55de148e11f448b966bc7f729d6c3a6 [term 1 LEADER]: Becoming Leader. State: Replica: e55de148e11f448b966bc7f729d6c3a6, State: Running, Role: LEADER
I20260924 03:06:44.605813 20904 consensus_queue.cc:237] T 00000000000000000000000000000000 P e55de148e11f448b966bc7f729d6c3a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e55de148e11f448b966bc7f729d6c3a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38159 } }
I20260924 03:06:44.605940 20894 sys_catalog.cc:564] T 00000000000000000000000000000000 P e55de148e11f448b966bc7f729d6c3a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:06:44.606933 20904 sys_catalog.cc:455] T 00000000000000000000000000000000 P e55de148e11f448b966bc7f729d6c3a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e55de148e11f448b966bc7f729d6c3a6. Latest consensus state: current_term: 1 leader_uuid: "e55de148e11f448b966bc7f729d6c3a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e55de148e11f448b966bc7f729d6c3a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38159 } } }
I20260924 03:06:44.607043 20904 sys_catalog.cc:458] T 00000000000000000000000000000000 P e55de148e11f448b966bc7f729d6c3a6 [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:44.607146 20905 sys_catalog.cc:455] T 00000000000000000000000000000000 P e55de148e11f448b966bc7f729d6c3a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e55de148e11f448b966bc7f729d6c3a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e55de148e11f448b966bc7f729d6c3a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38159 } } }
I20260924 03:06:44.607293 20905 sys_catalog.cc:458] T 00000000000000000000000000000000 P e55de148e11f448b966bc7f729d6c3a6 [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:44.607399 20911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:06:44.608131 20911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:06:44.614405 20911 catalog_manager.cc:1347] Generated new cluster ID: 66ef7bf25cb749c3b491e5afcdbc6902
I20260924 03:06:44.614542 20911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:06:44.632974 20911 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:06:44.634778 20911 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:06:44.644205 20911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e55de148e11f448b966bc7f729d6c3a6: Generated new TSK 0
I20260924 03:06:44.644506 20911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:06:44.652237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 20826
2026-09-24T03:06:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:06:44 dist-test-slave-wgsb krb5kdc[20925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:06:44 dist-test-slave-wgsb krb5kdc[20925](info): set up 2 sockets
Sep 24 03:06:44 dist-test-slave-wgsb krb5kdc[20925](info): commencing operation
krb5kdc: starting...
W20260924 03:06:45.735771 392 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:06:45 dist-test-slave-wgsb krb5kdc[20925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:06:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:06:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:06:45.863070 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46807
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:06:45.943344 20938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:45.943588 20938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:45.943638 20938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:45.944796 20938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:06:45.944865 20938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:45.944890 20938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:45.944917 20938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:06:45.944949 20938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:06:45.946444 20938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:45.946722 20938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:45.946949 20938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:45.951892 20944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:45.951925 20946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:45.952018 20938 server_base.cc:1034] running on GCE node
W20260924 03:06:45.951895 20943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:45.952351 20938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:45.952584 20938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:45.953737 20938 hybrid_clock.cc:648] HybridClock initialized: now 1790219205953729 us; error 31 us; skew 500 ppm
Sep 24 03:06:45 dist-test-slave-wgsb krb5kdc[20925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:45.956283 20938 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:06:45.956705 20938 webserver.cc:460] Webserver started at http://127.0.0.1:44485/ using document root <none> and password file <none>
I20260924 03:06:45.956940 20938 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:45.957010 20938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:45.957161 20938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:45.958220 20938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "77ce5791b42d444ca87bf1fd37961292"
format_stamp: "Formatted at 2026-09-24 03:06:45 on dist-test-slave-wgsb"
server_key: "0eb2a827d519b87d7ea3e4609c5c1e7b"
server_key_iv: "7a220d71f238f5a5faa3691f1f4b1b7c"
server_key_version: "encryptionkey@0"
I20260924 03:06:45.958598 20938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "77ce5791b42d444ca87bf1fd37961292"
format_stamp: "Formatted at 2026-09-24 03:06:45 on dist-test-slave-wgsb"
server_key: "0eb2a827d519b87d7ea3e4609c5c1e7b"
server_key_iv: "7a220d71f238f5a5faa3691f1f4b1b7c"
server_key_version: "encryptionkey@0"
I20260924 03:06:45.960044 20938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:06:45.960940 20953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:45.961236 20938 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:06:45.961344 20938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "77ce5791b42d444ca87bf1fd37961292"
format_stamp: "Formatted at 2026-09-24 03:06:45 on dist-test-slave-wgsb"
server_key: "0eb2a827d519b87d7ea3e4609c5c1e7b"
server_key_iv: "7a220d71f238f5a5faa3691f1f4b1b7c"
server_key_version: "encryptionkey@0"
I20260924 03:06:45.961419 20938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:45.977192 20938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:45.980906 20938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:45.981043 20938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:45.984746 20938 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46807
I20260924 03:06:45.984777 21005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46807 every 8 connection(s)
I20260924 03:06:45.985335 20938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:06:45.986178 21006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:45.988241 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 20938
I20260924 03:06:45.988428 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:06:45.988618 392 external_mini_cluster.cc:1428] Setting key 2498820dff3392575489ce4ab6763451
I20260924 03:06:45.989147 21006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77ce5791b42d444ca87bf1fd37961292: Bootstrap starting.
I20260924 03:06:45.989990 21006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77ce5791b42d444ca87bf1fd37961292: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:45.990321 21006 log.cc:824] T 00000000000000000000000000000000 P 77ce5791b42d444ca87bf1fd37961292: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:45.991321 21006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77ce5791b42d444ca87bf1fd37961292: No bootstrap required, opened a new log
I20260924 03:06:45.993698 21006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77ce5791b42d444ca87bf1fd37961292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77ce5791b42d444ca87bf1fd37961292" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46807 } }
I20260924 03:06:45.993872 21006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77ce5791b42d444ca87bf1fd37961292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:45.993906 21006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77ce5791b42d444ca87bf1fd37961292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ce5791b42d444ca87bf1fd37961292, State: Initialized, Role: FOLLOWER
I20260924 03:06:45.994052 21006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77ce5791b42d444ca87bf1fd37961292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77ce5791b42d444ca87bf1fd37961292" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46807 } }
I20260924 03:06:45.994154 21006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77ce5791b42d444ca87bf1fd37961292 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:06:45.994189 21006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77ce5791b42d444ca87bf1fd37961292 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:06:45.994227 21006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77ce5791b42d444ca87bf1fd37961292 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:45.995002 21006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77ce5791b42d444ca87bf1fd37961292 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77ce5791b42d444ca87bf1fd37961292" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46807 } }
I20260924 03:06:45.995100 21006 leader_election.cc:302] T 00000000000000000000000000000000 P 77ce5791b42d444ca87bf1fd37961292 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77ce5791b42d444ca87bf1fd37961292; no voters:
I20260924 03:06:45.995266 21006 leader_election.cc:288] T 00000000000000000000000000000000 P 77ce5791b42d444ca87bf1fd37961292 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:06:45.995342 21011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77ce5791b42d444ca87bf1fd37961292 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:06:45.995507 21011 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77ce5791b42d444ca87bf1fd37961292 [term 1 LEADER]: Becoming Leader. State: Replica: 77ce5791b42d444ca87bf1fd37961292, State: Running, Role: LEADER
I20260924 03:06:45.995545 21006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77ce5791b42d444ca87bf1fd37961292 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:06:45.995652 21011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77ce5791b42d444ca87bf1fd37961292 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77ce5791b42d444ca87bf1fd37961292" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46807 } }
I20260924 03:06:45.996050 21011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77ce5791b42d444ca87bf1fd37961292 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77ce5791b42d444ca87bf1fd37961292. Latest consensus state: current_term: 1 leader_uuid: "77ce5791b42d444ca87bf1fd37961292" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77ce5791b42d444ca87bf1fd37961292" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46807 } } }
I20260924 03:06:45.996320 21011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77ce5791b42d444ca87bf1fd37961292 [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:45.996246 21012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77ce5791b42d444ca87bf1fd37961292 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77ce5791b42d444ca87bf1fd37961292" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77ce5791b42d444ca87bf1fd37961292" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46807 } } }
I20260924 03:06:45.996604 21012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77ce5791b42d444ca87bf1fd37961292 [sys.catalog]: This master's current role is: LEADER
Sep 24 03:06:45 dist-test-slave-wgsb krb5kdc[20925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
W20260924 03:06:45.997408 21027 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 77ce5791b42d444ca87bf1fd37961292: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:06:45.997465 21027 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:06:45.997519 21021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:06:45.997705 21021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:06:46.000756 21009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:45.989178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57365 (local address 127.0.0.1:46807)
0924 03:06:45.989406 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:45.989416 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:45.989444 (+ 28us) server_negotiation.cc:407] Connection header received
0924 03:06:45.990179 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:45.990189 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:45.990274 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:45.990433 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:45.991261 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:45.992701 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:45.993487 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:45.994087 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:45.994200 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:45.997263 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:45.997267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:45.997267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:06:45.997278 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:45.999201 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:45.999640 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:45.999643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:45.999643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:45.999733 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:45.999944 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:45.999945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:45.999945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:46.000180 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:46.000366 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:46.000440 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 03:06:46.000588 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":78,"threads_started":1}
I20260924 03:06:46.001722 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:06:46.008196 21021 catalog_manager.cc:1347] Generated new cluster ID: 2f64af27d9814f798360cce1855695a4
I20260924 03:06:46.008256 21021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:06:46.015056 21009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:46.007958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33981 (local address 127.0.0.1:46807)
0924 03:06:46.007996 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:46.007998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:46.008230 (+ 232us) server_negotiation.cc:407] Connection header received
0924 03:06:46.008309 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:46.008310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:46.008321 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:46.008369 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:46.008911 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:46.009482 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:46.010203 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:46.010426 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:46.010505 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:46.012400 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:46.012401 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:46.012402 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:06:46.012409 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:46.013885 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:46.014251 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:46.014252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:46.014252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:46.014308 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:46.014510 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:46.014511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:46.014511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:46.014628 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:46.014728 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:46.014858 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 03:06:46.014934 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:06:46.030337 21021 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:06:46.030987 21021 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:06:46.042266 21021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77ce5791b42d444ca87bf1fd37961292: Generated new TSK 0
I20260924 03:06:46.042505 21021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:06:46.053376 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 20938
2026-09-24T03:06:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1398 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:06:46 dist-test-slave-wgsb krb5kdc[21037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:06:46 dist-test-slave-wgsb krb5kdc[21037](info): set up 2 sockets
Sep 24 03:06:46 dist-test-slave-wgsb krb5kdc[21037](info): commencing operation
krb5kdc: starting...
W20260924 03:06:47.184578 392 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:06:47 dist-test-slave-wgsb krb5kdc[21037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:06:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:06:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:06:47.341176 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35243
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:06:47.449822 21050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:47.450083 21050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:47.450114 21050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:47.451541 21050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:06:47.451668 21050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:47.451704 21050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:47.451717 21050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:06:47.451730 21050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:06:47.453687 21050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:47.453940 21050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:47.454137 21050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:47.459672 21058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:47.459537 21056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:47.459594 21055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:47.460150 21050 server_base.cc:1034] running on GCE node
I20260924 03:06:47.460450 21050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:47.460716 21050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:47.461907 21050 hybrid_clock.cc:648] HybridClock initialized: now 1790219207461883 us; error 37 us; skew 500 ppm
Sep 24 03:06:47 dist-test-slave-wgsb krb5kdc[21037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:47.471976 21050 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:06:47.472492 21050 webserver.cc:460] Webserver started at http://127.0.0.1:42477/ using document root <none> and password file <none>
I20260924 03:06:47.472756 21050 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:47.472826 21050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:47.472991 21050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:47.474134 21050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "391d2d8e4367435cb9f72e07b4764920"
format_stamp: "Formatted at 2026-09-24 03:06:47 on dist-test-slave-wgsb"
server_key: "3dd3f63355766d55045dd2b85c917560"
server_key_iv: "5139a4107e41664035e62e9c12a1be0d"
server_key_version: "encryptionkey@0"
I20260924 03:06:47.474520 21050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "391d2d8e4367435cb9f72e07b4764920"
format_stamp: "Formatted at 2026-09-24 03:06:47 on dist-test-slave-wgsb"
server_key: "3dd3f63355766d55045dd2b85c917560"
server_key_iv: "5139a4107e41664035e62e9c12a1be0d"
server_key_version: "encryptionkey@0"
I20260924 03:06:47.476986 21050 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 03:06:47.477880 21065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:47.478092 21050 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:06:47.478287 21050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "391d2d8e4367435cb9f72e07b4764920"
format_stamp: "Formatted at 2026-09-24 03:06:47 on dist-test-slave-wgsb"
server_key: "3dd3f63355766d55045dd2b85c917560"
server_key_iv: "5139a4107e41664035e62e9c12a1be0d"
server_key_version: "encryptionkey@0"
I20260924 03:06:47.478381 21050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:47.493232 21050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:47.495780 21050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:47.495958 21050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:47.501199 21050 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35243
I20260924 03:06:47.501302 21117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35243 every 8 connection(s)
I20260924 03:06:47.501914 21050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:06:47.502887 21118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:47.505436 21118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 391d2d8e4367435cb9f72e07b4764920: Bootstrap starting.
I20260924 03:06:47.506071 21118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 391d2d8e4367435cb9f72e07b4764920: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:47.506381 21118 log.cc:824] T 00000000000000000000000000000000 P 391d2d8e4367435cb9f72e07b4764920: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:47.507110 21118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 391d2d8e4367435cb9f72e07b4764920: No bootstrap required, opened a new log
I20260924 03:06:47.508952 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 21050
I20260924 03:06:47.509070 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:06:47.509030 21118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 391d2d8e4367435cb9f72e07b4764920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391d2d8e4367435cb9f72e07b4764920" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35243 } }
I20260924 03:06:47.509223 21118 raft_consensus.cc:374] T 00000000000000000000000000000000 P 391d2d8e4367435cb9f72e07b4764920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:47.509241 392 external_mini_cluster.cc:1428] Setting key 17f9dc197f5c477f2e77f89276bb5f4a
I20260924 03:06:47.509259 21118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 391d2d8e4367435cb9f72e07b4764920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391d2d8e4367435cb9f72e07b4764920, State: Initialized, Role: FOLLOWER
I20260924 03:06:47.509409 21118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 391d2d8e4367435cb9f72e07b4764920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391d2d8e4367435cb9f72e07b4764920" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35243 } }
I20260924 03:06:47.509528 21118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 391d2d8e4367435cb9f72e07b4764920 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:06:47.509573 21118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 391d2d8e4367435cb9f72e07b4764920 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:06:47.509658 21118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 391d2d8e4367435cb9f72e07b4764920 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:47.510592 21118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 391d2d8e4367435cb9f72e07b4764920 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391d2d8e4367435cb9f72e07b4764920" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35243 } }
I20260924 03:06:47.510731 21118 leader_election.cc:302] T 00000000000000000000000000000000 P 391d2d8e4367435cb9f72e07b4764920 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 391d2d8e4367435cb9f72e07b4764920; no voters:
I20260924 03:06:47.510928 21118 leader_election.cc:288] T 00000000000000000000000000000000 P 391d2d8e4367435cb9f72e07b4764920 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:06:47.510977 21123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 391d2d8e4367435cb9f72e07b4764920 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:06:47.511133 21123 raft_consensus.cc:686] T 00000000000000000000000000000000 P 391d2d8e4367435cb9f72e07b4764920 [term 1 LEADER]: Becoming Leader. State: Replica: 391d2d8e4367435cb9f72e07b4764920, State: Running, Role: LEADER
I20260924 03:06:47.511222 21118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 391d2d8e4367435cb9f72e07b4764920 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:06:47.511284 21123 consensus_queue.cc:237] T 00000000000000000000000000000000 P 391d2d8e4367435cb9f72e07b4764920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391d2d8e4367435cb9f72e07b4764920" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35243 } }
I20260924 03:06:47.511790 21125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391d2d8e4367435cb9f72e07b4764920 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "391d2d8e4367435cb9f72e07b4764920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391d2d8e4367435cb9f72e07b4764920" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35243 } } }
I20260924 03:06:47.511904 21125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391d2d8e4367435cb9f72e07b4764920 [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:47.511790 21124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391d2d8e4367435cb9f72e07b4764920 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 391d2d8e4367435cb9f72e07b4764920. Latest consensus state: current_term: 1 leader_uuid: "391d2d8e4367435cb9f72e07b4764920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391d2d8e4367435cb9f72e07b4764920" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35243 } } }
I20260924 03:06:47.511952 21124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391d2d8e4367435cb9f72e07b4764920 [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:47.512152 21132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:06:47.512883 21132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:06:47.514746 21132 catalog_manager.cc:1347] Generated new cluster ID: 1c39148b239e4a9895843e85602e68e8
I20260924 03:06:47.514813 21132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:06:47.515480 21122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:47.509812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55689 (local address 127.0.0.1:35243)
0924 03:06:47.510025 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:47.510036 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:47.510070 (+ 34us) server_negotiation.cc:407] Connection header received
0924 03:06:47.510194 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:47.510201 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:47.510246 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:47.510401 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:47.510610 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:47.512071 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:47.513273 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:47.513847 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:47.513987 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:47.514732 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:47.514736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:47.514737 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:47.514973 (+ 236us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:47.515011 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:47.515027 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:47.515090 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:47.515198 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 03:06:47.515260 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":90,"threads_started":1}
I20260924 03:06:47.516412 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:06:47.524546 21122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:47.517975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43329 (local address 127.0.0.1:35243)
0924 03:06:47.518015 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:47.518017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:47.519927 (+ 1910us) server_negotiation.cc:407] Connection header received
0924 03:06:47.519993 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:47.519994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:47.520008 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:47.520074 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:47.520221 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:47.520959 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:47.522256 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:47.522824 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:47.522936 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:47.523973 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:47.523976 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:47.523976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:47.523982 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:47.523997 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:47.524003 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:47.524031 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:47.524196 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 03:06:47.524424 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:06:47.532608 21132 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:06:47.533429 21132 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:06:47.539438 21132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 391d2d8e4367435cb9f72e07b4764920: Generated new TSK 0
I20260924 03:06:47.539678 21132 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:06:47.545420 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 21050
2026-09-24T03:06:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1494 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:06:47 dist-test-slave-wgsb krb5kdc[21149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:06:47 dist-test-slave-wgsb krb5kdc[21149](info): set up 2 sockets
Sep 24 03:06:47 dist-test-slave-wgsb krb5kdc[21149](info): commencing operation
krb5kdc: starting...
W20260924 03:06:48.683807 392 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:06:48 dist-test-slave-wgsb krb5kdc[21149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:06:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:06:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:06:48.811791 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46093
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:06:48.895787 21162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:48.896062 21162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:48.896103 21162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:48.897508 21162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:06:48.897614 21162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:48.897642 21162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:48.897675 21162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:06:48.897698 21162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:06:48.899320 21162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:48.899675 21162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:48.899930 21162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:48.905617 21170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:48.905613 21167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:48.905613 21168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:48.905925 21162 server_base.cc:1034] running on GCE node
I20260924 03:06:48.906155 21162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:48.906459 21162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:48.907660 21162 hybrid_clock.cc:648] HybridClock initialized: now 1790219208907654 us; error 39 us; skew 500 ppm
Sep 24 03:06:48 dist-test-slave-wgsb krb5kdc[21149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219208, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:48.910315 21162 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:06:48.910832 21162 webserver.cc:460] Webserver started at http://127.0.0.1:42537/ using document root <none> and password file <none>
I20260924 03:06:48.911070 21162 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:48.911140 21162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:48.911303 21162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:48.912477 21162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "58f5cf6221414a59a4865d917de9c6ee"
format_stamp: "Formatted at 2026-09-24 03:06:48 on dist-test-slave-wgsb"
server_key: "41f67efeb6ab448915ef5200ac52ad23"
server_key_iv: "dd79bc64f776e48bffd6fa84115900d9"
server_key_version: "encryptionkey@0"
I20260924 03:06:48.912869 21162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "58f5cf6221414a59a4865d917de9c6ee"
format_stamp: "Formatted at 2026-09-24 03:06:48 on dist-test-slave-wgsb"
server_key: "41f67efeb6ab448915ef5200ac52ad23"
server_key_iv: "dd79bc64f776e48bffd6fa84115900d9"
server_key_version: "encryptionkey@0"
I20260924 03:06:48.914332 21162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:06:48.915155 21177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:48.915371 21162 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:06:48.915490 21162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "58f5cf6221414a59a4865d917de9c6ee"
format_stamp: "Formatted at 2026-09-24 03:06:48 on dist-test-slave-wgsb"
server_key: "41f67efeb6ab448915ef5200ac52ad23"
server_key_iv: "dd79bc64f776e48bffd6fa84115900d9"
server_key_version: "encryptionkey@0"
I20260924 03:06:48.915576 21162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:48.920696 21162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:48.923002 21162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:48.923178 21162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:48.927660 21162 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46093
I20260924 03:06:48.927742 21229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46093 every 8 connection(s)
I20260924 03:06:48.928107 21162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:06:48.929178 21230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:48.931870 21230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58f5cf6221414a59a4865d917de9c6ee: Bootstrap starting.
I20260924 03:06:48.932508 21230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58f5cf6221414a59a4865d917de9c6ee: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:48.932821 21230 log.cc:824] T 00000000000000000000000000000000 P 58f5cf6221414a59a4865d917de9c6ee: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:48.933578 21230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58f5cf6221414a59a4865d917de9c6ee: No bootstrap required, opened a new log
I20260924 03:06:48.935213 21230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58f5cf6221414a59a4865d917de9c6ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58f5cf6221414a59a4865d917de9c6ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46093 } }
I20260924 03:06:48.935412 21230 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58f5cf6221414a59a4865d917de9c6ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:48.935437 21230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58f5cf6221414a59a4865d917de9c6ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f5cf6221414a59a4865d917de9c6ee, State: Initialized, Role: FOLLOWER
I20260924 03:06:48.935627 21230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58f5cf6221414a59a4865d917de9c6ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58f5cf6221414a59a4865d917de9c6ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46093 } }
I20260924 03:06:48.935726 21230 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58f5cf6221414a59a4865d917de9c6ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:06:48.935779 21230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58f5cf6221414a59a4865d917de9c6ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:06:48.935832 21230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58f5cf6221414a59a4865d917de9c6ee [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:48.936411 21230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58f5cf6221414a59a4865d917de9c6ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58f5cf6221414a59a4865d917de9c6ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46093 } }
I20260924 03:06:48.936496 21230 leader_election.cc:302] T 00000000000000000000000000000000 P 58f5cf6221414a59a4865d917de9c6ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58f5cf6221414a59a4865d917de9c6ee; no voters:
I20260924 03:06:48.936710 21230 leader_election.cc:288] T 00000000000000000000000000000000 P 58f5cf6221414a59a4865d917de9c6ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:06:48.936803 21233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58f5cf6221414a59a4865d917de9c6ee [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:06:48.936987 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 21162
I20260924 03:06:48.937016 21230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58f5cf6221414a59a4865d917de9c6ee [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:06:48.937024 21233 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58f5cf6221414a59a4865d917de9c6ee [term 1 LEADER]: Becoming Leader. State: Replica: 58f5cf6221414a59a4865d917de9c6ee, State: Running, Role: LEADER
I20260924 03:06:48.937098 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:06:48.937126 21233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58f5cf6221414a59a4865d917de9c6ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58f5cf6221414a59a4865d917de9c6ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46093 } }
I20260924 03:06:48.937276 392 external_mini_cluster.cc:1428] Setting key 6bdc54d49c816ea33fc5782a86788709
I20260924 03:06:48.937614 21234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58f5cf6221414a59a4865d917de9c6ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58f5cf6221414a59a4865d917de9c6ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58f5cf6221414a59a4865d917de9c6ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46093 } } }
I20260924 03:06:48.937685 21234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58f5cf6221414a59a4865d917de9c6ee [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:48.937991 21235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58f5cf6221414a59a4865d917de9c6ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58f5cf6221414a59a4865d917de9c6ee. Latest consensus state: current_term: 1 leader_uuid: "58f5cf6221414a59a4865d917de9c6ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58f5cf6221414a59a4865d917de9c6ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46093 } } }
I20260924 03:06:48.938051 21243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:06:48.938051 21235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58f5cf6221414a59a4865d917de9c6ee [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:48.938791 21243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:06:48.940434 21246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:48.937971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49993 (local address 127.0.0.1:46093)
0924 03:06:48.938134 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:48.938155 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:48.938465 (+ 310us) server_negotiation.cc:407] Connection header received
0924 03:06:48.938675 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:48.938683 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:48.938725 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:48.938885 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:48.939950 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:48.939953 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:48.939954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:48.939958 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:48.940008 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:48.940025 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:48.940101 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:48.940192 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 03:06:48.940248 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":68,"threads_started":1}
I20260924 03:06:48.940559 21243 catalog_manager.cc:1347] Generated new cluster ID: c4adf1fcf44e4510be1c44e00493caef
I20260924 03:06:48.940608 21243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:06:48.941290 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:06:48.944381 21243 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:06:48.944558 21243 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:06:48.951323 21243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58f5cf6221414a59a4865d917de9c6ee: Generated new TSK 0
I20260924 03:06:48.951435 21246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:48.943199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36295 (local address 127.0.0.1:46093)
0924 03:06:48.943243 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:48.943245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:48.943349 (+ 104us) server_negotiation.cc:407] Connection header received
0924 03:06:48.943410 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:48.943410 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:48.943425 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:48.943499 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:48.951050 (+ 7551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:48.951052 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:48.951053 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:48.951063 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:48.951089 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:48.951096 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:48.951182 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:48.951318 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 03:06:48.951335 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:06:48.951541 21243 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:06:48.954124 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 21162
2026-09-24T03:06:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:06:49 dist-test-slave-wgsb krb5kdc[21261](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:06:49 dist-test-slave-wgsb krb5kdc[21261](info): set up 2 sockets
Sep 24 03:06:49 dist-test-slave-wgsb krb5kdc[21261](info): commencing operation
krb5kdc: starting...
W20260924 03:06:50.056674 392 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:06:50 dist-test-slave-wgsb krb5kdc[21261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:06:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:06:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:06:50.200022 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42111
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:06:50.287971 21274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:50.288264 21274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:50.288309 21274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:50.289893 21274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:06:50.289964 21274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:50.289992 21274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:50.290025 21274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:06:50.290047 21274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:06:50.291723 21274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:50.292035 21274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:50.292289 21274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:50.297775 21280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:50.297775 21282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:50.297780 21279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:50.297996 21274 server_base.cc:1034] running on GCE node
I20260924 03:06:50.298259 21274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:50.298575 21274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:50.299690 21274 hybrid_clock.cc:648] HybridClock initialized: now 1790219210299648 us; error 46 us; skew 500 ppm
Sep 24 03:06:50 dist-test-slave-wgsb krb5kdc[21261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:50.302416 21274 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:06:50.302917 21274 webserver.cc:460] Webserver started at http://127.0.0.1:44527/ using document root <none> and password file <none>
I20260924 03:06:50.303164 21274 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:50.303239 21274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:50.303390 21274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:50.304548 21274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "5ef7a5dfa09d48a8b1293588348c672a"
format_stamp: "Formatted at 2026-09-24 03:06:50 on dist-test-slave-wgsb"
server_key: "51c65a59f3a73c2f04d3bda75445e387"
server_key_iv: "293a96abe314cbd398fc09cf66798834"
server_key_version: "encryptionkey@0"
I20260924 03:06:50.304924 21274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "5ef7a5dfa09d48a8b1293588348c672a"
format_stamp: "Formatted at 2026-09-24 03:06:50 on dist-test-slave-wgsb"
server_key: "51c65a59f3a73c2f04d3bda75445e387"
server_key_iv: "293a96abe314cbd398fc09cf66798834"
server_key_version: "encryptionkey@0"
I20260924 03:06:50.306538 21274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:06:50.307461 21289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:50.307693 21274 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:06:50.307801 21274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "5ef7a5dfa09d48a8b1293588348c672a"
format_stamp: "Formatted at 2026-09-24 03:06:50 on dist-test-slave-wgsb"
server_key: "51c65a59f3a73c2f04d3bda75445e387"
server_key_iv: "293a96abe314cbd398fc09cf66798834"
server_key_version: "encryptionkey@0"
I20260924 03:06:50.307904 21274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:50.331569 21274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:50.334045 21274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:50.334333 21274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:50.338873 21274 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42111
I20260924 03:06:50.338934 21341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42111 every 8 connection(s)
I20260924 03:06:50.339360 21274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:06:50.340317 21342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:50.342847 21342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ef7a5dfa09d48a8b1293588348c672a: Bootstrap starting.
I20260924 03:06:50.343387 21342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ef7a5dfa09d48a8b1293588348c672a: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:50.343711 21342 log.cc:824] T 00000000000000000000000000000000 P 5ef7a5dfa09d48a8b1293588348c672a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:50.344472 21342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ef7a5dfa09d48a8b1293588348c672a: No bootstrap required, opened a new log
I20260924 03:06:50.345549 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 21274
I20260924 03:06:50.345736 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:06:50.345947 392 external_mini_cluster.cc:1428] Setting key 7bec7073d98d16052ef9978d7e6fc9ad
I20260924 03:06:50.346347 21342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ef7a5dfa09d48a8b1293588348c672a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef7a5dfa09d48a8b1293588348c672a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42111 } }
I20260924 03:06:50.346539 21342 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ef7a5dfa09d48a8b1293588348c672a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:50.346628 21342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ef7a5dfa09d48a8b1293588348c672a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ef7a5dfa09d48a8b1293588348c672a, State: Initialized, Role: FOLLOWER
I20260924 03:06:50.346792 21342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ef7a5dfa09d48a8b1293588348c672a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef7a5dfa09d48a8b1293588348c672a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42111 } }
I20260924 03:06:50.346890 21342 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ef7a5dfa09d48a8b1293588348c672a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:06:50.346922 21342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ef7a5dfa09d48a8b1293588348c672a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:06:50.346961 21342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ef7a5dfa09d48a8b1293588348c672a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:50.347762 21342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ef7a5dfa09d48a8b1293588348c672a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef7a5dfa09d48a8b1293588348c672a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42111 } }
I20260924 03:06:50.347908 21342 leader_election.cc:302] T 00000000000000000000000000000000 P 5ef7a5dfa09d48a8b1293588348c672a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ef7a5dfa09d48a8b1293588348c672a; no voters:
I20260924 03:06:50.348094 21342 leader_election.cc:288] T 00000000000000000000000000000000 P 5ef7a5dfa09d48a8b1293588348c672a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:06:50.348116 21347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ef7a5dfa09d48a8b1293588348c672a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:06:50.348254 21347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ef7a5dfa09d48a8b1293588348c672a [term 1 LEADER]: Becoming Leader. State: Replica: 5ef7a5dfa09d48a8b1293588348c672a, State: Running, Role: LEADER
I20260924 03:06:50.348379 21342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ef7a5dfa09d48a8b1293588348c672a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:06:50.348356 21347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ef7a5dfa09d48a8b1293588348c672a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef7a5dfa09d48a8b1293588348c672a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42111 } }
I20260924 03:06:50.348863 21349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ef7a5dfa09d48a8b1293588348c672a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ef7a5dfa09d48a8b1293588348c672a. Latest consensus state: current_term: 1 leader_uuid: "5ef7a5dfa09d48a8b1293588348c672a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef7a5dfa09d48a8b1293588348c672a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42111 } } }
I20260924 03:06:50.348953 21349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ef7a5dfa09d48a8b1293588348c672a [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:50.349170 21348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ef7a5dfa09d48a8b1293588348c672a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ef7a5dfa09d48a8b1293588348c672a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef7a5dfa09d48a8b1293588348c672a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42111 } } }
I20260924 03:06:50.349291 21348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ef7a5dfa09d48a8b1293588348c672a [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:50.349457 21354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:06:50.350057 21354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:06:50.351997 21354 catalog_manager.cc:1347] Generated new cluster ID: 3f5899c3c9a54f12959604f472f19aa5
I20260924 03:06:50.352059 21354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:06:50 dist-test-slave-wgsb krb5kdc[21261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 03:06:50.358182 21346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:50.346508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:48581 (local address 127.0.0.1:42111)
0924 03:06:50.346809 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:50.346818 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:50.346852 (+ 34us) server_negotiation.cc:407] Connection header received
0924 03:06:50.346990 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:50.346999 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:50.347061 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:50.347217 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:50.348068 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:50.349707 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:50.350871 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:50.351512 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:50.351699 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:50.354402 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:50.354408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:50.354408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:06:50.354425 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:50.356529 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:50.356985 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:50.356991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:50.356991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:50.357097 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:50.357342 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:50.357343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:50.357343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:50.357597 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:50.357776 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:50.357901 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 03:06:50.358005 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":180,"threads_started":1}
I20260924 03:06:50.359319 21354 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:06:50.359398 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:06:50.360064 21354 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:06:50.366124 21354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ef7a5dfa09d48a8b1293588348c672a: Generated new TSK 0
I20260924 03:06:50.366324 21354 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:06:50.370847 21346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:50.360899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:37597 (local address 127.0.0.1:42111)
0924 03:06:50.360943 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:50.360946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:50.363844 (+ 2898us) server_negotiation.cc:407] Connection header received
0924 03:06:50.363925 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:50.363926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:50.363937 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:50.363989 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:50.364546 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:50.365063 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:50.365871 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:50.366262 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:50.366333 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:50.368013 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:50.368014 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:50.368015 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:06:50.368022 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:50.369560 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:50.369961 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:50.369963 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:50.369963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:50.370023 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:50.370332 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:50.370333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:50.370334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:50.370469 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:50.370576 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:50.370678 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 03:06:50.370742 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:06:50.373278 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 21274
W20260924 03:06:50.378834 21259 connection.cc:535] client connection to 127.0.0.1:42111 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42111): Connection reset by peer (error 104)
2026-09-24T03:06:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:06:50 dist-test-slave-wgsb krb5kdc[21373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:06:50 dist-test-slave-wgsb krb5kdc[21373](info): set up 2 sockets
Sep 24 03:06:50 dist-test-slave-wgsb krb5kdc[21373](info): commencing operation
krb5kdc: starting...
W20260924 03:06:51.456629 392 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:06:51 dist-test-slave-wgsb krb5kdc[21373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:06:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:06:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:06:51.587536 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37425
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:06:51.708877 21386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:51.709144 21386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:51.709175 21386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:51.710528 21386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:06:51.710588 21386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:51.710603 21386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:51.710616 21386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:06:51.710629 21386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:06:51.712354 21386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:51.712621 21386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:51.712823 21386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:51.720038 21391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:51.720177 21394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:51.720177 21392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:51.721190 21386 server_base.cc:1034] running on GCE node
I20260924 03:06:51.721462 21386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:51.721969 21386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:51.725359 21386 hybrid_clock.cc:648] HybridClock initialized: now 1790219211725170 us; error 198 us; skew 500 ppm
Sep 24 03:06:51 dist-test-slave-wgsb krb5kdc[21373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:51.729377 21386 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:06:51.729959 21386 webserver.cc:460] Webserver started at http://127.0.0.1:34421/ using document root <none> and password file <none>
I20260924 03:06:51.730237 21386 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:51.730309 21386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:51.730466 21386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:51.731695 21386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "457c2c5624224e5f8eefa3aab9e1eca8"
format_stamp: "Formatted at 2026-09-24 03:06:51 on dist-test-slave-wgsb"
server_key: "670d444e1f833c8c4a34bdb3455fa6a0"
server_key_iv: "91f64067fdc724c522d338004ca241d0"
server_key_version: "encryptionkey@0"
I20260924 03:06:51.732081 21386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "457c2c5624224e5f8eefa3aab9e1eca8"
format_stamp: "Formatted at 2026-09-24 03:06:51 on dist-test-slave-wgsb"
server_key: "670d444e1f833c8c4a34bdb3455fa6a0"
server_key_iv: "91f64067fdc724c522d338004ca241d0"
server_key_version: "encryptionkey@0"
I20260924 03:06:51.733577 21386 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:06:51.734445 21401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:51.734654 21386 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 03:06:51.734757 21386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "457c2c5624224e5f8eefa3aab9e1eca8"
format_stamp: "Formatted at 2026-09-24 03:06:51 on dist-test-slave-wgsb"
server_key: "670d444e1f833c8c4a34bdb3455fa6a0"
server_key_iv: "91f64067fdc724c522d338004ca241d0"
server_key_version: "encryptionkey@0"
I20260924 03:06:51.734910 21386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:51.749956 21386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:51.752254 21386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:51.752485 21386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:51.756875 21386 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37425
I20260924 03:06:51.756913 21453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37425 every 8 connection(s)
I20260924 03:06:51.757402 21386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:06:51.758438 21454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:51.760886 21454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 457c2c5624224e5f8eefa3aab9e1eca8: Bootstrap starting.
I20260924 03:06:51.761433 21454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 457c2c5624224e5f8eefa3aab9e1eca8: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:51.761664 21454 log.cc:824] T 00000000000000000000000000000000 P 457c2c5624224e5f8eefa3aab9e1eca8: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:51.762413 21454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 457c2c5624224e5f8eefa3aab9e1eca8: No bootstrap required, opened a new log
I20260924 03:06:51.764117 21454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 457c2c5624224e5f8eefa3aab9e1eca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457c2c5624224e5f8eefa3aab9e1eca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37425 } }
I20260924 03:06:51.764256 21454 raft_consensus.cc:374] T 00000000000000000000000000000000 P 457c2c5624224e5f8eefa3aab9e1eca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:51.764279 21454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 457c2c5624224e5f8eefa3aab9e1eca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 457c2c5624224e5f8eefa3aab9e1eca8, State: Initialized, Role: FOLLOWER
I20260924 03:06:51.764390 21454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 457c2c5624224e5f8eefa3aab9e1eca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457c2c5624224e5f8eefa3aab9e1eca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37425 } }
I20260924 03:06:51.764444 21454 raft_consensus.cc:388] T 00000000000000000000000000000000 P 457c2c5624224e5f8eefa3aab9e1eca8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:06:51.764464 21454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 457c2c5624224e5f8eefa3aab9e1eca8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:06:51.764492 21454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 457c2c5624224e5f8eefa3aab9e1eca8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:51.765026 21454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 457c2c5624224e5f8eefa3aab9e1eca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457c2c5624224e5f8eefa3aab9e1eca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37425 } }
I20260924 03:06:51.765115 21454 leader_election.cc:302] T 00000000000000000000000000000000 P 457c2c5624224e5f8eefa3aab9e1eca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 457c2c5624224e5f8eefa3aab9e1eca8; no voters:
I20260924 03:06:51.765241 21454 leader_election.cc:288] T 00000000000000000000000000000000 P 457c2c5624224e5f8eefa3aab9e1eca8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:06:51.765323 21457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 457c2c5624224e5f8eefa3aab9e1eca8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:06:51.765506 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 21386
I20260924 03:06:51.765537 21454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 457c2c5624224e5f8eefa3aab9e1eca8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:06:51.765582 21457 raft_consensus.cc:686] T 00000000000000000000000000000000 P 457c2c5624224e5f8eefa3aab9e1eca8 [term 1 LEADER]: Becoming Leader. State: Replica: 457c2c5624224e5f8eefa3aab9e1eca8, State: Running, Role: LEADER
I20260924 03:06:51.765635 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:06:51.765854 392 external_mini_cluster.cc:1428] Setting key 4d276e6435a916a6601e97996f758c8a
I20260924 03:06:51.765801 21457 consensus_queue.cc:237] T 00000000000000000000000000000000 P 457c2c5624224e5f8eefa3aab9e1eca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457c2c5624224e5f8eefa3aab9e1eca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37425 } }
I20260924 03:06:51.766232 21458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457c2c5624224e5f8eefa3aab9e1eca8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "457c2c5624224e5f8eefa3aab9e1eca8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457c2c5624224e5f8eefa3aab9e1eca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37425 } } }
I20260924 03:06:51.766315 21458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457c2c5624224e5f8eefa3aab9e1eca8 [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:51.766331 21459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457c2c5624224e5f8eefa3aab9e1eca8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 457c2c5624224e5f8eefa3aab9e1eca8. Latest consensus state: current_term: 1 leader_uuid: "457c2c5624224e5f8eefa3aab9e1eca8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457c2c5624224e5f8eefa3aab9e1eca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37425 } } }
I20260924 03:06:51.766417 21459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457c2c5624224e5f8eefa3aab9e1eca8 [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:51.766516 21463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:06:51.767109 21463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:06:51.771116 21463 catalog_manager.cc:1347] Generated new cluster ID: 0974df98826741b89ff1056fb50a047b
I20260924 03:06:51.771214 21463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:06:51.774622 21468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:51.766530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:60261 (local address 127.0.0.1:37425)
0924 03:06:51.769249 (+ 2719us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:51.769262 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:51.769318 (+ 56us) server_negotiation.cc:407] Connection header received
0924 03:06:51.769923 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:51.769939 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:51.770028 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:51.770221 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:51.770724 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:51.771874 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:51.772723 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:51.773138 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:51.773268 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:51.774012 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:51.774016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:51.774017 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:51.774204 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:51.774234 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:51.774245 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:51.774338 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:51.774415 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 03:06:51.774445 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2268,"thread_start_us":96,"threads_started":1}
I20260924 03:06:51.775579 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:06:51.782196 21463 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:06:51.782903 21463 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:06:51.785655 21468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:51.782042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:33139 (local address 127.0.0.1:37425)
0924 03:06:51.782096 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:51.782099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:51.782216 (+ 117us) server_negotiation.cc:407] Connection header received
0924 03:06:51.782273 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:51.782274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:51.782290 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:51.782368 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:51.782524 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:51.783200 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:51.784252 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:51.784550 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:51.784605 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:51.785371 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:51.785372 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:51.785373 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:51.785384 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:51.785399 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:51.785403 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:51.785461 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:51.785561 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 03:06:51.785573 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 03:06:51.787683 21463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 457c2c5624224e5f8eefa3aab9e1eca8: Generated new TSK 0
I20260924 03:06:51.787878 21463 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:06:51.793735 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 21386
W20260924 03:06:51.798347 21371 connection.cc:535] client connection to 127.0.0.1:37425 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37425): Connection reset by peer (error 104)
2026-09-24T03:06:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1419 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:06:51 dist-test-slave-wgsb krb5kdc[21485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:06:51 dist-test-slave-wgsb krb5kdc[21485](info): set up 2 sockets
Sep 24 03:06:51 dist-test-slave-wgsb krb5kdc[21485](info): commencing operation
krb5kdc: starting...
W20260924 03:06:52.908062 392 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:06:53 dist-test-slave-wgsb krb5kdc[21485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:06:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:06:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:06:53.064798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43631
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:06:53.158322 21498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:53.158557 21498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:53.158581 21498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:53.159869 21498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:06:53.159924 21498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:53.159940 21498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:53.159952 21498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:06:53.159965 21498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:06:53.161576 21498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:53.161809 21498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:53.162278 21498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:06:53.167946 21498 server_base.cc:1034] running on GCE node
W20260924 03:06:53.167894 21506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:53.167905 21503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:53.167889 21504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:53.168479 21498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:53.168879 21498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:53.170073 21498 hybrid_clock.cc:648] HybridClock initialized: now 1790219213170016 us; error 75 us; skew 500 ppm
Sep 24 03:06:53 dist-test-slave-wgsb krb5kdc[21485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:53.181130 21498 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:06:53.181764 21498 webserver.cc:460] Webserver started at http://127.0.0.1:33199/ using document root <none> and password file <none>
I20260924 03:06:53.182014 21498 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:53.182085 21498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:53.182245 21498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:53.183331 21498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "7719116308374c84832d40e47c68f47d"
format_stamp: "Formatted at 2026-09-24 03:06:53 on dist-test-slave-wgsb"
server_key: "a5eeb40d568d09afafe7070b7c0de46a"
server_key_iv: "c5d2c5067235eba8eb0a71d316322551"
server_key_version: "encryptionkey@0"
I20260924 03:06:53.183768 21498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "7719116308374c84832d40e47c68f47d"
format_stamp: "Formatted at 2026-09-24 03:06:53 on dist-test-slave-wgsb"
server_key: "a5eeb40d568d09afafe7070b7c0de46a"
server_key_iv: "c5d2c5067235eba8eb0a71d316322551"
server_key_version: "encryptionkey@0"
I20260924 03:06:53.185204 21498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:06:53.186133 21513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:53.186404 21498 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:06:53.186530 21498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "7719116308374c84832d40e47c68f47d"
format_stamp: "Formatted at 2026-09-24 03:06:53 on dist-test-slave-wgsb"
server_key: "a5eeb40d568d09afafe7070b7c0de46a"
server_key_iv: "c5d2c5067235eba8eb0a71d316322551"
server_key_version: "encryptionkey@0"
I20260924 03:06:53.186643 21498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:53.190757 21498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:53.193454 21498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:53.193660 21498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:53.198755 21498 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43631
I20260924 03:06:53.198917 21565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43631 every 8 connection(s)
I20260924 03:06:53.199373 21498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:06:53.200711 21566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:53.201088 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 21498
I20260924 03:06:53.201222 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:06:53.201445 392 external_mini_cluster.cc:1428] Setting key 8fc49e277ca7238585cd2d215627ce40
I20260924 03:06:53.204581 21569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:53.202040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:44141 (local address 127.0.0.1:43631)
0924 03:06:53.202230 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:53.202237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:53.202264 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:06:53.202705 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:53.202712 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:53.202738 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:53.202863 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:53.204033 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:53.204037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:53.204037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:53.204041 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:53.204082 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:53.204093 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:53.204155 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:53.204283 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 03:06:53.204323 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260924 03:06:53.205953 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:06:53.205972 21566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7719116308374c84832d40e47c68f47d: Bootstrap starting.
I20260924 03:06:53.206771 21566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7719116308374c84832d40e47c68f47d: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:53.207115 21566 log.cc:824] T 00000000000000000000000000000000 P 7719116308374c84832d40e47c68f47d: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:53.208480 21566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7719116308374c84832d40e47c68f47d: No bootstrap required, opened a new log
I20260924 03:06:53.210479 21566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7719116308374c84832d40e47c68f47d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7719116308374c84832d40e47c68f47d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43631 } }
I20260924 03:06:53.210634 21566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7719116308374c84832d40e47c68f47d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:53.210659 21566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7719116308374c84832d40e47c68f47d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7719116308374c84832d40e47c68f47d, State: Initialized, Role: FOLLOWER
I20260924 03:06:53.210808 21566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7719116308374c84832d40e47c68f47d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7719116308374c84832d40e47c68f47d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43631 } }
I20260924 03:06:53.210909 21566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7719116308374c84832d40e47c68f47d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:06:53.210944 21566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7719116308374c84832d40e47c68f47d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:06:53.211031 21566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7719116308374c84832d40e47c68f47d [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:53.211930 21566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7719116308374c84832d40e47c68f47d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7719116308374c84832d40e47c68f47d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43631 } }
I20260924 03:06:53.212085 21566 leader_election.cc:302] T 00000000000000000000000000000000 P 7719116308374c84832d40e47c68f47d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7719116308374c84832d40e47c68f47d; no voters:
I20260924 03:06:53.212332 21569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:53.207742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:41881 (local address 127.0.0.1:43631)
0924 03:06:53.207780 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:53.207782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:53.210955 (+ 3173us) server_negotiation.cc:407] Connection header received
0924 03:06:53.211053 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:53.211054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:53.211061 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:53.211130 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:53.211970 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:53.211972 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:53.211972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:53.212019 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:53.212039 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:53.212046 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:53.212115 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:53.212189 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 03:06:53.212279 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:06:53.212383 21566 leader_election.cc:288] T 00000000000000000000000000000000 P 7719116308374c84832d40e47c68f47d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:06:53.212450 21576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7719116308374c84832d40e47c68f47d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:06:53.212646 21576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7719116308374c84832d40e47c68f47d [term 1 LEADER]: Becoming Leader. State: Replica: 7719116308374c84832d40e47c68f47d, State: Running, Role: LEADER
I20260924 03:06:53.212677 21566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7719116308374c84832d40e47c68f47d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:06:53.212827 21576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7719116308374c84832d40e47c68f47d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7719116308374c84832d40e47c68f47d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43631 } }
I20260924 03:06:53.213208 21576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7719116308374c84832d40e47c68f47d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7719116308374c84832d40e47c68f47d. Latest consensus state: current_term: 1 leader_uuid: "7719116308374c84832d40e47c68f47d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7719116308374c84832d40e47c68f47d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43631 } } }
I20260924 03:06:53.213222 21577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7719116308374c84832d40e47c68f47d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7719116308374c84832d40e47c68f47d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7719116308374c84832d40e47c68f47d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43631 } } }
I20260924 03:06:53.213303 21576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7719116308374c84832d40e47c68f47d [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:53.213431 21577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7719116308374c84832d40e47c68f47d [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:53.213697 21584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:06:53.214293 21584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:06:53.216005 21584 catalog_manager.cc:1347] Generated new cluster ID: 4a6572e27c894330bff989982091f8b3
I20260924 03:06:53.216081 21584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:06:53.242736 21584 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:06:53.242964 21584 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:06:53.250953 21584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7719116308374c84832d40e47c68f47d: Generated new TSK 0
I20260924 03:06:53.251179 21584 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:06:53.255888 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 21498
2026-09-24T03:06:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1462 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:06:53 dist-test-slave-wgsb krb5kdc[21597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:06:53 dist-test-slave-wgsb krb5kdc[21597](info): set up 2 sockets
Sep 24 03:06:53 dist-test-slave-wgsb krb5kdc[21597](info): commencing operation
krb5kdc: starting...
W20260924 03:06:54.432456 392 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:06:54 dist-test-slave-wgsb krb5kdc[21597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:06:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:06:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:06:54.580376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42805
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:06:54.664551 21610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:54.664832 21610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:54.664880 21610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:54.666150 21610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:06:54.666235 21610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:54.666262 21610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:54.666296 21610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:06:54.666317 21610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:06:54.668082 21610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42805
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:54.668438 21610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:54.668682 21610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:54.674075 21615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:54.674091 21618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:54.674124 21610 server_base.cc:1034] running on GCE node
W20260924 03:06:54.674091 21616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:54.674633 21610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:54.674971 21610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:54.676118 21610 hybrid_clock.cc:648] HybridClock initialized: now 1790219214676068 us; error 60 us; skew 500 ppm
Sep 24 03:06:54 dist-test-slave-wgsb krb5kdc[21597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:54.679158 21610 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:06:54.679754 21610 webserver.cc:460] Webserver started at http://127.0.0.1:38447/ using document root <none> and password file <none>
I20260924 03:06:54.679996 21610 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:54.680045 21610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:54.680176 21610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:54.681336 21610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "f7ad4f3846e049238241effd0d37cf54"
format_stamp: "Formatted at 2026-09-24 03:06:54 on dist-test-slave-wgsb"
server_key: "01699401146b66a1ea333f9d41353e6b"
server_key_iv: "5afce655c186264ea35a25de44ec3d38"
server_key_version: "encryptionkey@0"
I20260924 03:06:54.681710 21610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "f7ad4f3846e049238241effd0d37cf54"
format_stamp: "Formatted at 2026-09-24 03:06:54 on dist-test-slave-wgsb"
server_key: "01699401146b66a1ea333f9d41353e6b"
server_key_iv: "5afce655c186264ea35a25de44ec3d38"
server_key_version: "encryptionkey@0"
I20260924 03:06:54.683445 21610 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 03:06:54.684444 21625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:54.684643 21610 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:06:54.684741 21610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "f7ad4f3846e049238241effd0d37cf54"
format_stamp: "Formatted at 2026-09-24 03:06:54 on dist-test-slave-wgsb"
server_key: "01699401146b66a1ea333f9d41353e6b"
server_key_iv: "5afce655c186264ea35a25de44ec3d38"
server_key_version: "encryptionkey@0"
I20260924 03:06:54.684921 21610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:54.711766 21610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:54.714363 21610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:54.714495 21610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:54.718936 21610 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42805
I20260924 03:06:54.718967 21677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42805 every 8 connection(s)
I20260924 03:06:54.719532 21610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:06:54.720412 21678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:54.724647 21678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ad4f3846e049238241effd0d37cf54: Bootstrap starting.
I20260924 03:06:54.725514 21678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7ad4f3846e049238241effd0d37cf54: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:54.725823 21678 log.cc:824] T 00000000000000000000000000000000 P f7ad4f3846e049238241effd0d37cf54: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:54.726459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 21610
I20260924 03:06:54.726589 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:06:54.726789 392 external_mini_cluster.cc:1428] Setting key 2b43be2b3e414c8bc01915b76b1f1441
I20260924 03:06:54.726899 21678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ad4f3846e049238241effd0d37cf54: No bootstrap required, opened a new log
I20260924 03:06:54.728983 21678 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7ad4f3846e049238241effd0d37cf54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ad4f3846e049238241effd0d37cf54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42805 } }
I20260924 03:06:54.729162 21678 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7ad4f3846e049238241effd0d37cf54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:54.729243 21678 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7ad4f3846e049238241effd0d37cf54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ad4f3846e049238241effd0d37cf54, State: Initialized, Role: FOLLOWER
I20260924 03:06:54.729401 21678 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7ad4f3846e049238241effd0d37cf54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ad4f3846e049238241effd0d37cf54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42805 } }
I20260924 03:06:54.729516 21678 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7ad4f3846e049238241effd0d37cf54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:06:54.729553 21678 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7ad4f3846e049238241effd0d37cf54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:06:54.729606 21678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7ad4f3846e049238241effd0d37cf54 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:54.731104 21678 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7ad4f3846e049238241effd0d37cf54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ad4f3846e049238241effd0d37cf54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42805 } }
I20260924 03:06:54.731334 21678 leader_election.cc:302] T 00000000000000000000000000000000 P f7ad4f3846e049238241effd0d37cf54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7ad4f3846e049238241effd0d37cf54; no voters:
I20260924 03:06:54.731539 21678 leader_election.cc:288] T 00000000000000000000000000000000 P f7ad4f3846e049238241effd0d37cf54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:06:54.731675 21683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7ad4f3846e049238241effd0d37cf54 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:06:54.731827 21683 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7ad4f3846e049238241effd0d37cf54 [term 1 LEADER]: Becoming Leader. State: Replica: f7ad4f3846e049238241effd0d37cf54, State: Running, Role: LEADER
I20260924 03:06:54.731837 21678 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7ad4f3846e049238241effd0d37cf54 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:06:54.731992 21683 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7ad4f3846e049238241effd0d37cf54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ad4f3846e049238241effd0d37cf54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42805 } }
I20260924 03:06:54.732489 21684 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ad4f3846e049238241effd0d37cf54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7ad4f3846e049238241effd0d37cf54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ad4f3846e049238241effd0d37cf54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42805 } } }
I20260924 03:06:54.732589 21684 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ad4f3846e049238241effd0d37cf54 [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:54.732791 21685 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ad4f3846e049238241effd0d37cf54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7ad4f3846e049238241effd0d37cf54. Latest consensus state: current_term: 1 leader_uuid: "f7ad4f3846e049238241effd0d37cf54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ad4f3846e049238241effd0d37cf54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42805 } } }
I20260924 03:06:54.733273 21692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:06:54.733242 21685 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ad4f3846e049238241effd0d37cf54 [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:54.733980 21692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 03:06:54 dist-test-slave-wgsb krb5kdc[21597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 03:06:54.736109 21692 catalog_manager.cc:1347] Generated new cluster ID: 6b29709a4e524e059c0f0f0931003990
I20260924 03:06:54.736181 21692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:06:54.740504 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:54.727438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:57937 (local address 127.0.0.1:42805)
0924 03:06:54.727635 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:54.727648 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:54.727675 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:06:54.728966 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:54.728975 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:54.729035 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:54.729328 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:54.730072 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:54.731086 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:54.732203 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:54.732867 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:54.732960 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:54.736318 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:54.736323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:54.736323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:06:54.736337 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:54.738456 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:54.738962 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:54.738965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:54.738966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:54.739111 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:54.739364 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:54.739365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:54.739365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:54.739734 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:54.739914 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:54.740003 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 03:06:54.740131 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260924 03:06:54.742161 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:06:54.756392 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:54.743950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:54757 (local address 127.0.0.1:42805)
0924 03:06:54.744000 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:54.744003 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:54.748202 (+ 4199us) server_negotiation.cc:407] Connection header received
0924 03:06:54.748318 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:54.748320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:54.748334 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:54.748401 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:54.749506 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:54.750304 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:54.751375 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:54.751759 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:54.751826 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:54.753374 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:54.753375 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:54.753376 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:06:54.753384 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:54.755035 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:54.755503 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:54.755504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:54.755505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:54.755563 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:54.755839 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:54.755840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:54.755840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:54.755971 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:54.756074 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:54.756224 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 03:06:54.756288 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 03:06:54.760066 21692 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:06:54.760836 21692 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:06:54.767951 21692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7ad4f3846e049238241effd0d37cf54: Generated new TSK 0
I20260924 03:06:54.768193 21692 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:06:54.772910 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 21610
W20260924 03:06:54.781245 21595 connection.cc:535] client connection to 127.0.0.1:42805 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42805): Connection reset by peer (error 104)
2026-09-24T03:06:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1526 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:06:54 dist-test-slave-wgsb krb5kdc[21709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:06:54 dist-test-slave-wgsb krb5kdc[21709](info): set up 2 sockets
Sep 24 03:06:54 dist-test-slave-wgsb krb5kdc[21709](info): commencing operation
krb5kdc: starting...
W20260924 03:06:55.853127 392 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:06:55 dist-test-slave-wgsb krb5kdc[21709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:06:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:06:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:06:56.001487 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44367
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:06:56.095485 21722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:56.095808 21722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:56.095837 21722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:56.097031 21722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:06:56.097075 21722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:56.097090 21722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:56.097103 21722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:06:56.097115 21722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:06:56.099364 21722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44367
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:56.099758 21722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:56.100024 21722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:56.106073 21727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:56.106073 21730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:56.106086 21728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:56.106531 21722 server_base.cc:1034] running on GCE node
I20260924 03:06:56.106768 21722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:56.107059 21722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:56.108222 21722 hybrid_clock.cc:648] HybridClock initialized: now 1790219216108210 us; error 31 us; skew 500 ppm
Sep 24 03:06:56 dist-test-slave-wgsb krb5kdc[21709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:56.112636 21722 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:06:56.113194 21722 webserver.cc:460] Webserver started at http://127.0.0.1:33943/ using document root <none> and password file <none>
I20260924 03:06:56.113443 21722 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:56.113514 21722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:56.113673 21722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:56.114846 21722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "1c452294facf4e7ab2cfd8a3e925cdc4"
format_stamp: "Formatted at 2026-09-24 03:06:56 on dist-test-slave-wgsb"
server_key: "d5de2bb68fd141c5f4df9ebb15b756b9"
server_key_iv: "8bdaba978626a85b10272ec6ef754ac8"
server_key_version: "encryptionkey@0"
I20260924 03:06:56.115254 21722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "1c452294facf4e7ab2cfd8a3e925cdc4"
format_stamp: "Formatted at 2026-09-24 03:06:56 on dist-test-slave-wgsb"
server_key: "d5de2bb68fd141c5f4df9ebb15b756b9"
server_key_iv: "8bdaba978626a85b10272ec6ef754ac8"
server_key_version: "encryptionkey@0"
I20260924 03:06:56.116930 21722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:06:56.118410 21737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:56.118925 21722 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:06:56.119081 21722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "1c452294facf4e7ab2cfd8a3e925cdc4"
format_stamp: "Formatted at 2026-09-24 03:06:56 on dist-test-slave-wgsb"
server_key: "d5de2bb68fd141c5f4df9ebb15b756b9"
server_key_iv: "8bdaba978626a85b10272ec6ef754ac8"
server_key_version: "encryptionkey@0"
I20260924 03:06:56.119192 21722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:56.133034 21722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:56.135953 21722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:56.136170 21722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:56.140851 21722 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44367
I20260924 03:06:56.140880 21789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44367 every 8 connection(s)
I20260924 03:06:56.141323 21722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:06:56.142350 21790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:56.144663 21790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c452294facf4e7ab2cfd8a3e925cdc4: Bootstrap starting.
I20260924 03:06:56.145293 21790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c452294facf4e7ab2cfd8a3e925cdc4: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:56.145581 21790 log.cc:824] T 00000000000000000000000000000000 P 1c452294facf4e7ab2cfd8a3e925cdc4: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:56.146384 21790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c452294facf4e7ab2cfd8a3e925cdc4: No bootstrap required, opened a new log
I20260924 03:06:56.147516 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 21722
I20260924 03:06:56.147648 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:06:56.147840 392 external_mini_cluster.cc:1428] Setting key fff4019ca5fb6befdef5b4913f9d7c93
I20260924 03:06:56.148406 21790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c452294facf4e7ab2cfd8a3e925cdc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c452294facf4e7ab2cfd8a3e925cdc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44367 } }
I20260924 03:06:56.148620 21790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c452294facf4e7ab2cfd8a3e925cdc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:56.148672 21790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c452294facf4e7ab2cfd8a3e925cdc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c452294facf4e7ab2cfd8a3e925cdc4, State: Initialized, Role: FOLLOWER
I20260924 03:06:56.148820 21790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c452294facf4e7ab2cfd8a3e925cdc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c452294facf4e7ab2cfd8a3e925cdc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44367 } }
I20260924 03:06:56.148885 21790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c452294facf4e7ab2cfd8a3e925cdc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:06:56.148907 21790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c452294facf4e7ab2cfd8a3e925cdc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:06:56.148970 21790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c452294facf4e7ab2cfd8a3e925cdc4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:56.150274 21790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c452294facf4e7ab2cfd8a3e925cdc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c452294facf4e7ab2cfd8a3e925cdc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44367 } }
I20260924 03:06:56.150370 21790 leader_election.cc:302] T 00000000000000000000000000000000 P 1c452294facf4e7ab2cfd8a3e925cdc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c452294facf4e7ab2cfd8a3e925cdc4; no voters:
I20260924 03:06:56.150563 21790 leader_election.cc:288] T 00000000000000000000000000000000 P 1c452294facf4e7ab2cfd8a3e925cdc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:06:56.150584 21795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c452294facf4e7ab2cfd8a3e925cdc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:06:56.150702 21795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c452294facf4e7ab2cfd8a3e925cdc4 [term 1 LEADER]: Becoming Leader. State: Replica: 1c452294facf4e7ab2cfd8a3e925cdc4, State: Running, Role: LEADER
I20260924 03:06:56.150839 21790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c452294facf4e7ab2cfd8a3e925cdc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:06:56.150838 21795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c452294facf4e7ab2cfd8a3e925cdc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c452294facf4e7ab2cfd8a3e925cdc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44367 } }
I20260924 03:06:56.151377 21796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c452294facf4e7ab2cfd8a3e925cdc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c452294facf4e7ab2cfd8a3e925cdc4. Latest consensus state: current_term: 1 leader_uuid: "1c452294facf4e7ab2cfd8a3e925cdc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c452294facf4e7ab2cfd8a3e925cdc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44367 } } }
I20260924 03:06:56.151472 21796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c452294facf4e7ab2cfd8a3e925cdc4 [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:56.151729 21797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c452294facf4e7ab2cfd8a3e925cdc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c452294facf4e7ab2cfd8a3e925cdc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c452294facf4e7ab2cfd8a3e925cdc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44367 } } }
I20260924 03:06:56.151832 21797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c452294facf4e7ab2cfd8a3e925cdc4 [sys.catalog]: This master's current role is: LEADER
W20260924 03:06:56.152802 21811 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1c452294facf4e7ab2cfd8a3e925cdc4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:06:56.152858 21811 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:06:56.152904 21803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:06:56.153048 21803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:06:56.154709 21803 catalog_manager.cc:1347] Generated new cluster ID: 2beaa96942c5497b9ebb49dbfb38c6ba
I20260924 03:06:56.154762 21803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:06:56.154919 21794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:56.148404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:49249 (local address 127.0.0.1:44367)
0924 03:06:56.148609 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:56.148620 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:56.148661 (+ 41us) server_negotiation.cc:407] Connection header received
0924 03:06:56.148786 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:56.148794 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:56.148853 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:56.149038 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:56.149403 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:56.150835 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:56.151960 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:56.152553 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:56.152691 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:56.154130 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:56.154134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:56.154134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:56.154420 (+ 286us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:56.154452 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:56.154464 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:56.154570 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:56.154655 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 03:06:56.154683 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":96,"threads_started":1}
I20260924 03:06:56.156201 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:06:56.161190 21803 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:06:56.162616 21803 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:06:56.164866 21794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:56.157577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:47437 (local address 127.0.0.1:44367)
0924 03:06:56.157618 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:56.157620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:56.160196 (+ 2576us) server_negotiation.cc:407] Connection header received
0924 03:06:56.160467 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:56.160469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:56.160484 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:56.160545 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:56.160794 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:56.161578 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:56.163249 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:56.163572 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:56.163712 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:56.164512 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:56.164514 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:56.164514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:56.164520 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:56.164538 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:56.164542 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:56.164608 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:56.164756 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 03:06:56.164772 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:06:56.172068 21803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c452294facf4e7ab2cfd8a3e925cdc4: Generated new TSK 0
I20260924 03:06:56.172297 21803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:06:56.175477 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 21722
W20260924 03:06:56.179463 21707 connection.cc:535] client connection to 127.0.0.1:44367 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44367): Connection reset by peer (error 104)
2026-09-24T03:06:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1392 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:06:56 dist-test-slave-wgsb krb5kdc[21821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:06:56 dist-test-slave-wgsb krb5kdc[21821](info): set up 2 sockets
Sep 24 03:06:56 dist-test-slave-wgsb krb5kdc[21821](info): commencing operation
krb5kdc: starting...
W20260924 03:06:57.396227 392 mini_kdc.cc:121] Time spent starting KDC: real 1.202s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:06:57 dist-test-slave-wgsb krb5kdc[21821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:06:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:06:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:06:57.532357 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41439
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:06:57.632470 21834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:57.632725 21834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:57.632756 21834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:57.634065 21834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:06:57.634109 21834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:57.634125 21834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:57.634140 21834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:06:57.634152 21834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:06:57.635820 21834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41439
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218948286999-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:57.636049 21834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:57.636263 21834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:06:57.642513 21834 server_base.cc:1034] running on GCE node
W20260924 03:06:57.642642 21842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:57.642614 21840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:57.642486 21839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:57.643232 21834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:57.643569 21834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:57.644776 21834 hybrid_clock.cc:648] HybridClock initialized: now 1790219217644740 us; error 51 us; skew 500 ppm
Sep 24 03:06:57 dist-test-slave-wgsb krb5kdc[21821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:57.647428 21834 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:06:57.648510 21834 webserver.cc:460] Webserver started at http://127.0.0.1:40819/ using document root <none> and password file <none>
I20260924 03:06:57.648874 21834 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:57.648962 21834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:57.649438 21834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:57.651352 21834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "9561540efc8a4f749f4c47097639c76a"
format_stamp: "Formatted at 2026-09-24 03:06:57 on dist-test-slave-wgsb"
server_key: "a44910545fc1bb2e6a1f0991bb771a33"
server_key_iv: "911677e09201f0f7cce29ea57f96c34e"
server_key_version: "encryptionkey@0"
I20260924 03:06:57.651980 21834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "9561540efc8a4f749f4c47097639c76a"
format_stamp: "Formatted at 2026-09-24 03:06:57 on dist-test-slave-wgsb"
server_key: "a44910545fc1bb2e6a1f0991bb771a33"
server_key_iv: "911677e09201f0f7cce29ea57f96c34e"
server_key_version: "encryptionkey@0"
I20260924 03:06:57.653643 21834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:06:57.654786 21849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:57.655042 21834 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:06:57.655143 21834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "9561540efc8a4f749f4c47097639c76a"
format_stamp: "Formatted at 2026-09-24 03:06:57 on dist-test-slave-wgsb"
server_key: "a44910545fc1bb2e6a1f0991bb771a33"
server_key_iv: "911677e09201f0f7cce29ea57f96c34e"
server_key_version: "encryptionkey@0"
I20260924 03:06:57.655243 21834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:57.658818 21834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:57.661615 21834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:57.661736 21834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:57.666612 21834 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41439
I20260924 03:06:57.666646 21901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41439 every 8 connection(s)
I20260924 03:06:57.667032 21834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:06:57.668213 21902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:57.669380 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 21834
I20260924 03:06:57.669504 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:06:57.669678 392 external_mini_cluster.cc:1428] Setting key 8e633a7e75eb9104403523bb915d3019
I20260924 03:06:57.671202 21902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9561540efc8a4f749f4c47097639c76a: Bootstrap starting.
I20260924 03:06:57.672008 21902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9561540efc8a4f749f4c47097639c76a: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:57.672243 21905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:57.670251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:58941 (local address 127.0.0.1:41439)
0924 03:06:57.670437 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:57.670445 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:57.670470 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:06:57.670891 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:57.670897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:57.670920 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:57.671035 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:57.671791 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:57.671794 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:57.671794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:57.671798 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:57.671834 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:57.671844 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:57.671913 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:57.671993 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 03:06:57.672046 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260924 03:06:57.672397 21902 log.cc:824] T 00000000000000000000000000000000 P 9561540efc8a4f749f4c47097639c76a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:57.673300 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:06:57.673360 21902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9561540efc8a4f749f4c47097639c76a: No bootstrap required, opened a new log
I20260924 03:06:57.675892 21902 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9561540efc8a4f749f4c47097639c76a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9561540efc8a4f749f4c47097639c76a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41439 } }
I20260924 03:06:57.676096 21902 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9561540efc8a4f749f4c47097639c76a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:57.676141 21902 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9561540efc8a4f749f4c47097639c76a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9561540efc8a4f749f4c47097639c76a, State: Initialized, Role: FOLLOWER
I20260924 03:06:57.676033 21905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:57.674806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:49017 (local address 127.0.0.1:41439)
0924 03:06:57.674844 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:57.674846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:57.674986 (+ 140us) server_negotiation.cc:407] Connection header received
0924 03:06:57.675065 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:57.675066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:57.675070 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:57.675111 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:57.675821 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:57.675822 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:57.675823 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:06:57.675827 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:57.675841 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:06:57.675845 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:57.675901 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:57.675955 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 03:06:57.675967 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:06:57.676311 21902 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9561540efc8a4f749f4c47097639c76a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9561540efc8a4f749f4c47097639c76a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41439 } }
I20260924 03:06:57.676425 21902 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9561540efc8a4f749f4c47097639c76a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:06:57.676465 21902 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9561540efc8a4f749f4c47097639c76a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:06:57.676512 21902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9561540efc8a4f749f4c47097639c76a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:57.677244 21902 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9561540efc8a4f749f4c47097639c76a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9561540efc8a4f749f4c47097639c76a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41439 } }
I20260924 03:06:57.677362 21902 leader_election.cc:302] T 00000000000000000000000000000000 P 9561540efc8a4f749f4c47097639c76a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9561540efc8a4f749f4c47097639c76a; no voters:
I20260924 03:06:57.677542 21902 leader_election.cc:288] T 00000000000000000000000000000000 P 9561540efc8a4f749f4c47097639c76a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:06:57.677639 21912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9561540efc8a4f749f4c47097639c76a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:06:57.677838 21912 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9561540efc8a4f749f4c47097639c76a [term 1 LEADER]: Becoming Leader. State: Replica: 9561540efc8a4f749f4c47097639c76a, State: Running, Role: LEADER
I20260924 03:06:57.677886 21902 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9561540efc8a4f749f4c47097639c76a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:06:57.678018 21912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9561540efc8a4f749f4c47097639c76a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9561540efc8a4f749f4c47097639c76a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41439 } }
I20260924 03:06:57.678468 21913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9561540efc8a4f749f4c47097639c76a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9561540efc8a4f749f4c47097639c76a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9561540efc8a4f749f4c47097639c76a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41439 } } }
I20260924 03:06:57.678557 21913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9561540efc8a4f749f4c47097639c76a [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:57.678723 21914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9561540efc8a4f749f4c47097639c76a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9561540efc8a4f749f4c47097639c76a. Latest consensus state: current_term: 1 leader_uuid: "9561540efc8a4f749f4c47097639c76a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9561540efc8a4f749f4c47097639c76a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41439 } } }
I20260924 03:06:57.678778 21914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9561540efc8a4f749f4c47097639c76a [sys.catalog]: This master's current role is: LEADER
W20260924 03:06:57.679739 21928 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9561540efc8a4f749f4c47097639c76a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:06:57.679807 21928 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:06:57.679878 21921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:06:57.680058 21921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:06:57.681815 21921 catalog_manager.cc:1347] Generated new cluster ID: 9d436df898ed48cd94a46769d0e3c298
I20260924 03:06:57.681869 21921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:06:57.710350 21921 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:06:57.710661 21921 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:06:57.717180 21921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9561540efc8a4f749f4c47097639c76a: Generated new TSK 0
I20260924 03:06:57.717391 21921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:06:57.725102 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 21834
2026-09-24T03:06:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1550 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:06:57 dist-test-slave-wgsb krb5kdc[21933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:06:57 dist-test-slave-wgsb krb5kdc[21933](info): set up 2 sockets
Sep 24 03:06:57 dist-test-slave-wgsb krb5kdc[21933](info): commencing operation
krb5kdc: starting...
W20260924 03:06:58.832345 392 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:06:58 dist-test-slave-wgsb krb5kdc[21933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:06:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:06:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:06:58.990833 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34063
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:06:59.097260 21946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:06:59.097565 21946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:06:59.097608 21946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:06:59.098901 21946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:06:59.098984 21946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:06:59.099009 21946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:06:59.099037 21946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:06:59.099068 21946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:06:59.100672 21946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34063
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:06:59.100986 21946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:06:59.101233 21946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:06:59.106868 21952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:59.106958 21954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:06:59.106887 21951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:06:59.106998 21946 server_base.cc:1034] running on GCE node
I20260924 03:06:59.107509 21946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:06:59.107928 21946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:06:59.109122 21946 hybrid_clock.cc:648] HybridClock initialized: now 1790219219109091 us; error 53 us; skew 500 ppm
Sep 24 03:06:59 dist-test-slave-wgsb krb5kdc[21933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:06:59.118505 21946 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:06:59.118965 21946 webserver.cc:460] Webserver started at http://127.0.98.62:41497/ using document root <none> and password file <none>
I20260924 03:06:59.119159 21946 fs_manager.cc:359] Metadata directory not provided
I20260924 03:06:59.119204 21946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:06:59.119297 21946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:06:59.120430 21946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "00741b8fe5aa42d79fd7ea9f21c7f39c"
format_stamp: "Formatted at 2026-09-24 03:06:59 on dist-test-slave-wgsb"
server_key: "2a641c5242c3efd1fa9a6e779baaa137"
server_key_iv: "810056997875561330aa19e6e40fb9dc"
server_key_version: "encryptionkey@0"
I20260924 03:06:59.120750 21946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "00741b8fe5aa42d79fd7ea9f21c7f39c"
format_stamp: "Formatted at 2026-09-24 03:06:59 on dist-test-slave-wgsb"
server_key: "2a641c5242c3efd1fa9a6e779baaa137"
server_key_iv: "810056997875561330aa19e6e40fb9dc"
server_key_version: "encryptionkey@0"
I20260924 03:06:59.122082 21946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:06:59.122968 21961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:06:59.123250 21946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:06:59.123390 21946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "00741b8fe5aa42d79fd7ea9f21c7f39c"
format_stamp: "Formatted at 2026-09-24 03:06:59 on dist-test-slave-wgsb"
server_key: "2a641c5242c3efd1fa9a6e779baaa137"
server_key_iv: "810056997875561330aa19e6e40fb9dc"
server_key_version: "encryptionkey@0"
I20260924 03:06:59.123487 21946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:06:59.156564 21946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:06:59.158990 21946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:06:59.159188 21946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:06:59.164371 21946 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34063
I20260924 03:06:59.164429 22013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34063 every 8 connection(s)
I20260924 03:06:59.164845 21946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:06:59.166317 22014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:06:59.168718 22014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00741b8fe5aa42d79fd7ea9f21c7f39c: Bootstrap starting.
I20260924 03:06:59.169265 22014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00741b8fe5aa42d79fd7ea9f21c7f39c: Neither blocks nor log segments found. Creating new log.
I20260924 03:06:59.169529 22014 log.cc:824] T 00000000000000000000000000000000 P 00741b8fe5aa42d79fd7ea9f21c7f39c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:06:59.169762 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 21946
I20260924 03:06:59.169890 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:06:59.170084 392 external_mini_cluster.cc:1428] Setting key 004e367868e9c5fbd0b0445db1808b1d
I20260924 03:06:59.170367 22014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00741b8fe5aa42d79fd7ea9f21c7f39c: No bootstrap required, opened a new log
I20260924 03:06:59.172659 22014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00741b8fe5aa42d79fd7ea9f21c7f39c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00741b8fe5aa42d79fd7ea9f21c7f39c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34063 } }
I20260924 03:06:59.172832 22014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00741b8fe5aa42d79fd7ea9f21c7f39c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:06:59.172869 22014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00741b8fe5aa42d79fd7ea9f21c7f39c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00741b8fe5aa42d79fd7ea9f21c7f39c, State: Initialized, Role: FOLLOWER
I20260924 03:06:59.173022 22014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00741b8fe5aa42d79fd7ea9f21c7f39c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00741b8fe5aa42d79fd7ea9f21c7f39c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34063 } }
I20260924 03:06:59.173110 22014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00741b8fe5aa42d79fd7ea9f21c7f39c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:06:59.173151 22014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00741b8fe5aa42d79fd7ea9f21c7f39c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:06:59.173192 22014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00741b8fe5aa42d79fd7ea9f21c7f39c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:06:59.173975 22014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00741b8fe5aa42d79fd7ea9f21c7f39c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00741b8fe5aa42d79fd7ea9f21c7f39c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34063 } }
I20260924 03:06:59.174095 22014 leader_election.cc:302] T 00000000000000000000000000000000 P 00741b8fe5aa42d79fd7ea9f21c7f39c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00741b8fe5aa42d79fd7ea9f21c7f39c; no voters:
I20260924 03:06:59.174283 22014 leader_election.cc:288] T 00000000000000000000000000000000 P 00741b8fe5aa42d79fd7ea9f21c7f39c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:06:59.174362 22019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00741b8fe5aa42d79fd7ea9f21c7f39c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:06:59.174538 22014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00741b8fe5aa42d79fd7ea9f21c7f39c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:06:59.174564 22019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00741b8fe5aa42d79fd7ea9f21c7f39c [term 1 LEADER]: Becoming Leader. State: Replica: 00741b8fe5aa42d79fd7ea9f21c7f39c, State: Running, Role: LEADER
I20260924 03:06:59.174753 22019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00741b8fe5aa42d79fd7ea9f21c7f39c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00741b8fe5aa42d79fd7ea9f21c7f39c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34063 } }
I20260924 03:06:59.175974 22021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00741b8fe5aa42d79fd7ea9f21c7f39c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00741b8fe5aa42d79fd7ea9f21c7f39c. Latest consensus state: current_term: 1 leader_uuid: "00741b8fe5aa42d79fd7ea9f21c7f39c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00741b8fe5aa42d79fd7ea9f21c7f39c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34063 } } }
I20260924 03:06:59.175956 22020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00741b8fe5aa42d79fd7ea9f21c7f39c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00741b8fe5aa42d79fd7ea9f21c7f39c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00741b8fe5aa42d79fd7ea9f21c7f39c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34063 } } }
I20260924 03:06:59.176075 22021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00741b8fe5aa42d79fd7ea9f21c7f39c [sys.catalog]: This master's current role is: LEADER
I20260924 03:06:59.176092 22020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00741b8fe5aa42d79fd7ea9f21c7f39c [sys.catalog]: This master's current role is: LEADER
W20260924 03:06:59.176446 22034 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 00741b8fe5aa42d79fd7ea9f21c7f39c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:06:59.178284 22034 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:06:59.178422 22035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:06:59.178649 22035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 03:06:59 dist-test-slave-wgsb krb5kdc[21933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:06:59.180383 22035 catalog_manager.cc:1347] Generated new cluster ID: e29e8118a47042e4a37cdef3f585d573
I20260924 03:06:59.180444 22035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:06:59.184733 22018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:59.170717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35367 (local address 127.0.98.62:34063)
0924 03:06:59.170926 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:59.170937 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:59.170972 (+ 35us) server_negotiation.cc:407] Connection header received
0924 03:06:59.171872 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:59.171883 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:59.171961 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:59.172121 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:59.172990 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:59.174754 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:59.176299 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:59.176752 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:59.176903 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:59.180550 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:59.180555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:59.180556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:06:59.180570 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:59.182631 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:59.183095 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:59.183099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:59.183099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:59.183214 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:59.183443 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:59.183444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:59.183445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:59.183790 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:59.183968 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:59.184407 (+ 439us) server_negotiation.cc:299] Negotiation successful
0924 03:06:59.184511 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":104,"threads_started":1}
I20260924 03:06:59.185792 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:06:59.191071 22035 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:06:59.191874 22035 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:06:59.195472 22018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:06:59.187361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39277 (local address 127.0.98.62:34063)
0924 03:06:59.187398 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 03:06:59.187400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:06:59.187880 (+ 480us) server_negotiation.cc:407] Connection header received
0924 03:06:59.187945 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:06:59.187946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:06:59.187957 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:06:59.188009 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:06:59.188643 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:59.189516 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:59.190571 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:06:59.190914 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:06:59.191000 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:06:59.192501 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:06:59.192503 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:06:59.192503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:06:59.192510 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:06:59.194217 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:59.194644 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:59.194645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:59.194645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:59.194705 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:06:59.194935 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:06:59.194935 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:06:59.194936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:06:59.195057 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:06:59.195157 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 03:06:59.195311 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 03:06:59.195376 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:06:59.201897 22035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00741b8fe5aa42d79fd7ea9f21c7f39c: Generated new TSK 0
I20260924 03:06:59.202168 22035 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:06:59.207688 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 21946
W20260924 03:06:59.212080 21931 connection.cc:535] client connection to 127.0.98.62:34063 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:34063): Connection reset by peer (error 104)
2026-09-24T03:06:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1481 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:06:59 dist-test-slave-wgsb krb5kdc[22045](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:06:59 dist-test-slave-wgsb krb5kdc[22045](info): set up 2 sockets
Sep 24 03:06:59 dist-test-slave-wgsb krb5kdc[22045](info): commencing operation
krb5kdc: starting...
W20260924 03:07:00.331919 392 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:07:00 dist-test-slave-wgsb krb5kdc[22045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:07:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:07:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:07:00.475646 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43301
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:07:00.561956 22058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:07:00.562203 22058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:07:00.562240 22058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:07:00.563445 22058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:07:00.563519 22058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:07:00.563544 22058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:07:00.563575 22058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:07:00.563622 22058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:07:00.565161 22058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43301
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:07:00.565455 22058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:07:00.565698 22058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:07:00.571158 22064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:07:00.571165 22063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:07:00.571401 22058 server_base.cc:1034] running on GCE node
W20260924 03:07:00.571516 22066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:07:00.571799 22058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:07:00.572064 22058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:07:00.573252 22058 hybrid_clock.cc:648] HybridClock initialized: now 1790219220573205 us; error 63 us; skew 500 ppm
Sep 24 03:07:00 dist-test-slave-wgsb krb5kdc[22045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:07:00.576140 22058 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:07:00.577775 22058 webserver.cc:460] Webserver started at http://127.0.98.62:39851/ using document root <none> and password file <none>
I20260924 03:07:00.578377 22058 fs_manager.cc:359] Metadata directory not provided
I20260924 03:07:00.578485 22058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:07:00.578657 22058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:07:00.579912 22058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "e3ee3e0b5b3f4502b3151e5e45728242"
format_stamp: "Formatted at 2026-09-24 03:07:00 on dist-test-slave-wgsb"
server_key: "06beed29dafdfb167d29c180069b865a"
server_key_iv: "9b28e5e9185f4ec7ca7521f3d073566a"
server_key_version: "encryptionkey@0"
I20260924 03:07:00.580292 22058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "e3ee3e0b5b3f4502b3151e5e45728242"
format_stamp: "Formatted at 2026-09-24 03:07:00 on dist-test-slave-wgsb"
server_key: "06beed29dafdfb167d29c180069b865a"
server_key_iv: "9b28e5e9185f4ec7ca7521f3d073566a"
server_key_version: "encryptionkey@0"
I20260924 03:07:00.581897 22058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:07:00.582742 22073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:07:00.582953 22058 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:07:00.583057 22058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "e3ee3e0b5b3f4502b3151e5e45728242"
format_stamp: "Formatted at 2026-09-24 03:07:00 on dist-test-slave-wgsb"
server_key: "06beed29dafdfb167d29c180069b865a"
server_key_iv: "9b28e5e9185f4ec7ca7521f3d073566a"
server_key_version: "encryptionkey@0"
I20260924 03:07:00.583148 22058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:07:00.625029 22058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:07:00.627229 22058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:07:00.627424 22058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:07:00.633071 22058 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43301
I20260924 03:07:00.633118 22125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43301 every 8 connection(s)
I20260924 03:07:00.633495 22058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:07:00.634410 22126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:07:00.636751 22126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3ee3e0b5b3f4502b3151e5e45728242: Bootstrap starting.
I20260924 03:07:00.637387 22126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3ee3e0b5b3f4502b3151e5e45728242: Neither blocks nor log segments found. Creating new log.
I20260924 03:07:00.637758 22126 log.cc:824] T 00000000000000000000000000000000 P e3ee3e0b5b3f4502b3151e5e45728242: Log is configured to *not* fsync() on all Append() calls
I20260924 03:07:00.638533 22126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3ee3e0b5b3f4502b3151e5e45728242: No bootstrap required, opened a new log
I20260924 03:07:00.640249 22126 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3ee3e0b5b3f4502b3151e5e45728242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ee3e0b5b3f4502b3151e5e45728242" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43301 } }
I20260924 03:07:00.640416 22126 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3ee3e0b5b3f4502b3151e5e45728242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:07:00.640441 22126 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3ee3e0b5b3f4502b3151e5e45728242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3ee3e0b5b3f4502b3151e5e45728242, State: Initialized, Role: FOLLOWER
I20260924 03:07:00.640592 22126 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3ee3e0b5b3f4502b3151e5e45728242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ee3e0b5b3f4502b3151e5e45728242" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43301 } }
I20260924 03:07:00.640668 22126 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3ee3e0b5b3f4502b3151e5e45728242 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:07:00.640691 22126 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3ee3e0b5b3f4502b3151e5e45728242 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:07:00.640720 22126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3ee3e0b5b3f4502b3151e5e45728242 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:07:00.641776 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 22058
I20260924 03:07:00.641929 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:07:00.642084 22126 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3ee3e0b5b3f4502b3151e5e45728242 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ee3e0b5b3f4502b3151e5e45728242" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43301 } }
I20260924 03:07:00.642261 392 external_mini_cluster.cc:1428] Setting key 2c94c703f0d7d13c5703ebaa2cb1ac70
I20260924 03:07:00.642292 22126 leader_election.cc:302] T 00000000000000000000000000000000 P e3ee3e0b5b3f4502b3151e5e45728242 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3ee3e0b5b3f4502b3151e5e45728242; no voters:
I20260924 03:07:00.642712 22126 leader_election.cc:288] T 00000000000000000000000000000000 P e3ee3e0b5b3f4502b3151e5e45728242 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:07:00.642725 22129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3ee3e0b5b3f4502b3151e5e45728242 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:07:00.643121 22126 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3ee3e0b5b3f4502b3151e5e45728242 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:07:00.643160 22129 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3ee3e0b5b3f4502b3151e5e45728242 [term 1 LEADER]: Becoming Leader. State: Replica: e3ee3e0b5b3f4502b3151e5e45728242, State: Running, Role: LEADER
I20260924 03:07:00.643347 22129 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3ee3e0b5b3f4502b3151e5e45728242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ee3e0b5b3f4502b3151e5e45728242" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43301 } }
I20260924 03:07:00.643889 22132 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ee3e0b5b3f4502b3151e5e45728242 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3ee3e0b5b3f4502b3151e5e45728242. Latest consensus state: current_term: 1 leader_uuid: "e3ee3e0b5b3f4502b3151e5e45728242" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ee3e0b5b3f4502b3151e5e45728242" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43301 } } }
I20260924 03:07:00.643994 22130 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ee3e0b5b3f4502b3151e5e45728242 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3ee3e0b5b3f4502b3151e5e45728242" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ee3e0b5b3f4502b3151e5e45728242" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43301 } } }
I20260924 03:07:00.644047 22132 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ee3e0b5b3f4502b3151e5e45728242 [sys.catalog]: This master's current role is: LEADER
I20260924 03:07:00.644052 22130 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ee3e0b5b3f4502b3151e5e45728242 [sys.catalog]: This master's current role is: LEADER
I20260924 03:07:00.644337 22140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:07:00.644956 22140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:07:00.646785 22140 catalog_manager.cc:1347] Generated new cluster ID: 9fcf41ea562e49b893d68d161b241fa2
I20260924 03:07:00.646842 22140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:07:00.651036 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:00.643170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44427 (local address 127.0.98.62:43301)
0924 03:07:00.643435 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:00.643451 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:00.644084 (+ 633us) server_negotiation.cc:407] Connection header received
0924 03:07:00.644223 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:00.644231 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:00.644297 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:00.644455 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:00.644861 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:00.647374 (+ 2513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:00.648542 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:00.649226 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:00.649357 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:00.650142 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:00.650147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:00.650147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:07:00.650488 (+ 341us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:00.650534 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:07:00.650551 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:00.650648 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:00.650751 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 03:07:00.650784 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":71,"threads_started":1}
I20260924 03:07:00.652137 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:07:00.655369 22140 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:07:00.656011 22140 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:07:00.657624 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:00.654038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33285 (local address 127.0.98.62:43301)
0924 03:07:00.654105 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:00.654108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:00.654181 (+ 73us) server_negotiation.cc:407] Connection header received
0924 03:07:00.654253 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:00.654254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:00.654265 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:00.654344 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:00.654553 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:00.655071 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:00.656190 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:00.656512 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:00.656579 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:00.657357 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:00.657358 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:00.657358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:07:00.657364 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:00.657381 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:07:00.657385 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:00.657442 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:00.657535 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 03:07:00.657547 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 03:07:00.664800 22140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3ee3e0b5b3f4502b3151e5e45728242: Generated new TSK 0
I20260924 03:07:00.665032 22140 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:07:00.672626 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 22058
W20260924 03:07:00.678589 22043 connection.cc:535] client connection to 127.0.98.62:43301 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:43301): Connection reset by peer (error 104)
2026-09-24T03:07:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1466 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:07:00 dist-test-slave-wgsb krb5kdc[22157](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:07:00 dist-test-slave-wgsb krb5kdc[22157](info): set up 2 sockets
Sep 24 03:07:00 dist-test-slave-wgsb krb5kdc[22157](info): commencing operation
krb5kdc: starting...
W20260924 03:07:01.739678 392 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:07:01 dist-test-slave-wgsb krb5kdc[22157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:07:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:07:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:07:01.863412 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45303
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:07:01.946557 22170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:07:01.946813 22170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:07:01.946843 22170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:07:01.948709 22170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:07:01.948767 22170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:07:01.948781 22170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:07:01.948796 22170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:07:01.948807 22170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:07:01.950440 22170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45303
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:07:01.950721 22170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:07:01.950974 22170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:07:01.956211 22178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:07:01.956235 22175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:07:01.956235 22176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:07:01.956673 22170 server_base.cc:1034] running on GCE node
I20260924 03:07:01.956997 22170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:07:01.957306 22170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:07:01.958498 22170 hybrid_clock.cc:648] HybridClock initialized: now 1790219221958486 us; error 29 us; skew 500 ppm
Sep 24 03:07:01 dist-test-slave-wgsb krb5kdc[22157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:07:01.961203 22170 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:07:01.961719 22170 webserver.cc:460] Webserver started at http://127.0.98.62:37541/ using document root <none> and password file <none>
I20260924 03:07:01.961969 22170 fs_manager.cc:359] Metadata directory not provided
I20260924 03:07:01.962038 22170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:07:01.962191 22170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:07:01.963282 22170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "1f0c20401c984746a0870f85d05374f7"
format_stamp: "Formatted at 2026-09-24 03:07:01 on dist-test-slave-wgsb"
server_key: "ee741f03b83146027e4f86113a8bee14"
server_key_iv: "e24ae19d43e34a66f82ab3dbcd521209"
server_key_version: "encryptionkey@0"
I20260924 03:07:01.963701 22170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "1f0c20401c984746a0870f85d05374f7"
format_stamp: "Formatted at 2026-09-24 03:07:01 on dist-test-slave-wgsb"
server_key: "ee741f03b83146027e4f86113a8bee14"
server_key_iv: "e24ae19d43e34a66f82ab3dbcd521209"
server_key_version: "encryptionkey@0"
I20260924 03:07:01.965112 22170 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:07:01.966035 22185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:07:01.966248 22170 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:07:01.966351 22170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "1f0c20401c984746a0870f85d05374f7"
format_stamp: "Formatted at 2026-09-24 03:07:01 on dist-test-slave-wgsb"
server_key: "ee741f03b83146027e4f86113a8bee14"
server_key_iv: "e24ae19d43e34a66f82ab3dbcd521209"
server_key_version: "encryptionkey@0"
I20260924 03:07:01.966440 22170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:07:01.969951 22170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:07:01.972157 22170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:07:01.972316 22170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:07:01.976560 22170 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45303
I20260924 03:07:01.976596 22237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45303 every 8 connection(s)
I20260924 03:07:01.976946 22170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:07:01.977990 22238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:07:01.978461 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 22170
I20260924 03:07:01.978596 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:07:01.978785 392 external_mini_cluster.cc:1428] Setting key c45e3529921b6c285465ac3b10a1c43e
I20260924 03:07:01.981375 22238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f0c20401c984746a0870f85d05374f7: Bootstrap starting.
I20260924 03:07:01.981549 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:01.979413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58471 (local address 127.0.98.62:45303)
0924 03:07:01.979579 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:01.979586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:01.979662 (+ 76us) server_negotiation.cc:407] Connection header received
0924 03:07:01.980079 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:01.980086 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:01.980110 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:01.980227 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:01.980988 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:01.980991 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:01.980992 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:07:01.980995 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:01.981042 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:07:01.981052 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:01.981107 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:01.981211 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 03:07:01.981242 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
I20260924 03:07:01.982120 22238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f0c20401c984746a0870f85d05374f7: Neither blocks nor log segments found. Creating new log.
I20260924 03:07:01.982422 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:07:01.982518 22238 log.cc:824] T 00000000000000000000000000000000 P 1f0c20401c984746a0870f85d05374f7: Log is configured to *not* fsync() on all Append() calls
I20260924 03:07:01.983258 22238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f0c20401c984746a0870f85d05374f7: No bootstrap required, opened a new log
I20260924 03:07:01.984903 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:01.983874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39489 (local address 127.0.98.62:45303)
0924 03:07:01.983912 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:01.983913 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:01.983986 (+ 73us) server_negotiation.cc:407] Connection header received
0924 03:07:01.984061 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:01.984061 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:01.984064 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:01.984109 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:01.984680 (+ 571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:01.984681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:01.984681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:07:01.984685 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:01.984699 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:07:01.984702 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:01.984741 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:01.984805 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 03:07:01.984816 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 03:07:01.985975 22238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f0c20401c984746a0870f85d05374f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f0c20401c984746a0870f85d05374f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45303 } }
I20260924 03:07:01.986147 22238 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f0c20401c984746a0870f85d05374f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:07:01.986198 22238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f0c20401c984746a0870f85d05374f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0c20401c984746a0870f85d05374f7, State: Initialized, Role: FOLLOWER
I20260924 03:07:01.986351 22238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f0c20401c984746a0870f85d05374f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f0c20401c984746a0870f85d05374f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45303 } }
I20260924 03:07:01.986454 22238 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f0c20401c984746a0870f85d05374f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:07:01.986505 22238 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f0c20401c984746a0870f85d05374f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:07:01.986557 22238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f0c20401c984746a0870f85d05374f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:07:01.987141 22238 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f0c20401c984746a0870f85d05374f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f0c20401c984746a0870f85d05374f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45303 } }
I20260924 03:07:01.987280 22238 leader_election.cc:302] T 00000000000000000000000000000000 P 1f0c20401c984746a0870f85d05374f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f0c20401c984746a0870f85d05374f7; no voters:
I20260924 03:07:01.987489 22238 leader_election.cc:288] T 00000000000000000000000000000000 P 1f0c20401c984746a0870f85d05374f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:07:01.987528 22248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f0c20401c984746a0870f85d05374f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:07:01.987731 22248 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f0c20401c984746a0870f85d05374f7 [term 1 LEADER]: Becoming Leader. State: Replica: 1f0c20401c984746a0870f85d05374f7, State: Running, Role: LEADER
I20260924 03:07:01.987794 22238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f0c20401c984746a0870f85d05374f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:07:01.989388 22248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f0c20401c984746a0870f85d05374f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f0c20401c984746a0870f85d05374f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45303 } }
I20260924 03:07:01.990063 22249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f0c20401c984746a0870f85d05374f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f0c20401c984746a0870f85d05374f7. Latest consensus state: current_term: 1 leader_uuid: "1f0c20401c984746a0870f85d05374f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f0c20401c984746a0870f85d05374f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45303 } } }
I20260924 03:07:01.990258 22250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f0c20401c984746a0870f85d05374f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f0c20401c984746a0870f85d05374f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f0c20401c984746a0870f85d05374f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45303 } } }
I20260924 03:07:01.990638 22249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f0c20401c984746a0870f85d05374f7 [sys.catalog]: This master's current role is: LEADER
I20260924 03:07:01.990639 22250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f0c20401c984746a0870f85d05374f7 [sys.catalog]: This master's current role is: LEADER
W20260924 03:07:01.991076 22262 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f0c20401c984746a0870f85d05374f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:07:01.991124 22262 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:07:01.991185 22264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:07:01.991354 22264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:07:01.993480 22264 catalog_manager.cc:1347] Generated new cluster ID: ecc6fc53e096474185ba532b6369a0ac
I20260924 03:07:01.993544 22264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:07:02.008558 22264 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:07:02.008775 22264 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:07:02.015632 22264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f0c20401c984746a0870f85d05374f7: Generated new TSK 0
I20260924 03:07:02.015813 22264 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:07:02.020001 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 22170
2026-09-24T03:07:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:07:02 dist-test-slave-wgsb krb5kdc[22269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:07:02 dist-test-slave-wgsb krb5kdc[22269](info): set up 2 sockets
Sep 24 03:07:02 dist-test-slave-wgsb krb5kdc[22269](info): commencing operation
krb5kdc: starting...
W20260924 03:07:03.084507 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:07:03 dist-test-slave-wgsb krb5kdc[22269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:07:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:07:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:07:03.210278 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39581
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:07:03.292122 22282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:07:03.292346 22282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:07:03.292371 22282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:07:03.293474 22282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:07:03.293511 22282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:07:03.293525 22282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:07:03.293537 22282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:07:03.293550 22282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:07:03.294945 22282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:07:03.295215 22282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:07:03.295475 22282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:07:03.300659 22290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:07:03.300581 22287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:07:03.300583 22288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:07:03.300917 22282 server_base.cc:1034] running on GCE node
I20260924 03:07:03.301156 22282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:07:03.301446 22282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:07:03.302614 22282 hybrid_clock.cc:648] HybridClock initialized: now 1790219223302602 us; error 29 us; skew 500 ppm
Sep 24 03:07:03 dist-test-slave-wgsb krb5kdc[22269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:07:03.305143 22282 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:07:03.305866 22282 webserver.cc:460] Webserver started at http://127.0.98.62:43887/ using document root <none> and password file <none>
I20260924 03:07:03.306133 22282 fs_manager.cc:359] Metadata directory not provided
I20260924 03:07:03.306206 22282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:07:03.306401 22282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:07:03.307788 22282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "032a531d0f274f88a87f474d280fb086"
format_stamp: "Formatted at 2026-09-24 03:07:03 on dist-test-slave-wgsb"
server_key: "b636be0e9588c61c4ab52f7bc03621d1"
server_key_iv: "3f26e06664e7fbb2af6af9b26613d3d0"
server_key_version: "encryptionkey@0"
I20260924 03:07:03.308164 22282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "032a531d0f274f88a87f474d280fb086"
format_stamp: "Formatted at 2026-09-24 03:07:03 on dist-test-slave-wgsb"
server_key: "b636be0e9588c61c4ab52f7bc03621d1"
server_key_iv: "3f26e06664e7fbb2af6af9b26613d3d0"
server_key_version: "encryptionkey@0"
I20260924 03:07:03.309635 22282 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:07:03.310515 22297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:07:03.310772 22282 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:07:03.310889 22282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "032a531d0f274f88a87f474d280fb086"
format_stamp: "Formatted at 2026-09-24 03:07:03 on dist-test-slave-wgsb"
server_key: "b636be0e9588c61c4ab52f7bc03621d1"
server_key_iv: "3f26e06664e7fbb2af6af9b26613d3d0"
server_key_version: "encryptionkey@0"
I20260924 03:07:03.310974 22282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:07:03.321663 22282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:07:03.328533 22282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:07:03.328727 22282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:07:03.333122 22282 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39581
I20260924 03:07:03.333213 22349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39581 every 8 connection(s)
I20260924 03:07:03.333575 22282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:07:03.334457 22350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:07:03.335915 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 22282
I20260924 03:07:03.336035 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:07:03.336213 392 external_mini_cluster.cc:1428] Setting key 9c1c9424bfa2ec36609f0551ea1c0bfb
I20260924 03:07:03.337218 22350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032a531d0f274f88a87f474d280fb086: Bootstrap starting.
I20260924 03:07:03.337918 22350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 032a531d0f274f88a87f474d280fb086: Neither blocks nor log segments found. Creating new log.
I20260924 03:07:03.338271 22350 log.cc:824] T 00000000000000000000000000000000 P 032a531d0f274f88a87f474d280fb086: Log is configured to *not* fsync() on all Append() calls
I20260924 03:07:03.339059 22350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032a531d0f274f88a87f474d280fb086: No bootstrap required, opened a new log
I20260924 03:07:03.341280 22350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 032a531d0f274f88a87f474d280fb086 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a531d0f274f88a87f474d280fb086" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39581 } }
I20260924 03:07:03.341418 22350 raft_consensus.cc:374] T 00000000000000000000000000000000 P 032a531d0f274f88a87f474d280fb086 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:07:03.341440 22350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 032a531d0f274f88a87f474d280fb086 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032a531d0f274f88a87f474d280fb086, State: Initialized, Role: FOLLOWER
I20260924 03:07:03.341599 22350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 032a531d0f274f88a87f474d280fb086 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a531d0f274f88a87f474d280fb086" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39581 } }
I20260924 03:07:03.341686 22350 raft_consensus.cc:388] T 00000000000000000000000000000000 P 032a531d0f274f88a87f474d280fb086 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:07:03.341732 22350 raft_consensus.cc:482] T 00000000000000000000000000000000 P 032a531d0f274f88a87f474d280fb086 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:07:03.341815 22350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 032a531d0f274f88a87f474d280fb086 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:07:03.342468 22350 raft_consensus.cc:504] T 00000000000000000000000000000000 P 032a531d0f274f88a87f474d280fb086 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a531d0f274f88a87f474d280fb086" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39581 } }
I20260924 03:07:03.342559 22350 leader_election.cc:302] T 00000000000000000000000000000000 P 032a531d0f274f88a87f474d280fb086 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 032a531d0f274f88a87f474d280fb086; no voters:
I20260924 03:07:03.342720 22350 leader_election.cc:288] T 00000000000000000000000000000000 P 032a531d0f274f88a87f474d280fb086 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:07:03.342793 22355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 032a531d0f274f88a87f474d280fb086 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:07:03.343030 22355 raft_consensus.cc:686] T 00000000000000000000000000000000 P 032a531d0f274f88a87f474d280fb086 [term 1 LEADER]: Becoming Leader. State: Replica: 032a531d0f274f88a87f474d280fb086, State: Running, Role: LEADER
I20260924 03:07:03.343040 22350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 032a531d0f274f88a87f474d280fb086 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:07:03.343257 22355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 032a531d0f274f88a87f474d280fb086 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a531d0f274f88a87f474d280fb086" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39581 } }
I20260924 03:07:03.343662 22355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032a531d0f274f88a87f474d280fb086 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 032a531d0f274f88a87f474d280fb086. Latest consensus state: current_term: 1 leader_uuid: "032a531d0f274f88a87f474d280fb086" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a531d0f274f88a87f474d280fb086" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39581 } } }
I20260924 03:07:03.343814 22355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032a531d0f274f88a87f474d280fb086 [sys.catalog]: This master's current role is: LEADER
I20260924 03:07:03.344347 22364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 24 03:07:03 dist-test-slave-wgsb krb5kdc[22269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:07:03.344739 22356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032a531d0f274f88a87f474d280fb086 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "032a531d0f274f88a87f474d280fb086" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a531d0f274f88a87f474d280fb086" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39581 } } }
I20260924 03:07:03.344817 22356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032a531d0f274f88a87f474d280fb086 [sys.catalog]: This master's current role is: LEADER
I20260924 03:07:03.344974 22364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:07:03.346899 22364 catalog_manager.cc:1347] Generated new cluster ID: 647b295e45c0438a8bc10f9200dc8387
I20260924 03:07:03.346956 22364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:07:03.349198 22353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:03.336789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46373 (local address 127.0.98.62:39581)
0924 03:07:03.336972 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:03.336980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:03.337005 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:07:03.337455 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:03.337461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:03.337509 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:03.337626 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:03.338509 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:03.339868 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:03.341002 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:03.341405 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:03.341550 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:03.345201 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:03.345206 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:03.345206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:03.345219 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:03.347415 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:03.347897 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:03.347900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:03.347901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:03.347998 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:03.348232 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:03.348233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:03.348233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:03.348495 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:03.348661 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:03.348810 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 03:07:03.348976 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":68,"threads_started":1}
I20260924 03:07:03.350232 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:07:03.362262 22364 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:07:03.363046 22364 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:07:03.367695 22353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:03.351934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38037 (local address 127.0.98.62:39581)
0924 03:07:03.351972 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:03.351975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:03.359915 (+ 7940us) server_negotiation.cc:407] Connection header received
0924 03:07:03.360026 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:03.360028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:03.360042 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:03.360154 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:03.360916 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:03.361480 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:03.362628 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:03.362884 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:03.362987 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:03.364497 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:03.364499 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:03.364500 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:03.364508 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:03.366233 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:03.366683 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:03.366684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:03.366684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:03.366753 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:03.366995 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:03.366996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:03.366997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:03.367126 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:03.367250 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:03.367398 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 03:07:03.367503 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:07:03.374100 22364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 032a531d0f274f88a87f474d280fb086: Generated new TSK 0
I20260924 03:07:03.374311 22364 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:07:03.378098 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 22282
2026-09-24T03:07:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:07:03 dist-test-slave-wgsb krb5kdc[22381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:07:03 dist-test-slave-wgsb krb5kdc[22381](info): set up 2 sockets
Sep 24 03:07:03 dist-test-slave-wgsb krb5kdc[22381](info): commencing operation
krb5kdc: starting...
W20260924 03:07:04.439811 392 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:07:04 dist-test-slave-wgsb krb5kdc[22381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:07:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:07:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:07:04.575420 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34341
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:07:04.660686 22394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:07:04.660907 22394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:07:04.660930 22394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:07:04.662128 22394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:07:04.662175 22394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:07:04.662190 22394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:07:04.662202 22394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:07:04.662214 22394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:07:04.663751 22394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:07:04.664028 22394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:07:04.664358 22394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:07:04.669761 22402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:07:04.669790 22400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:07:04.669889 22399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:07:04.670096 22394 server_base.cc:1034] running on GCE node
I20260924 03:07:04.670394 22394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:07:04.670754 22394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:07:04.671913 22394 hybrid_clock.cc:648] HybridClock initialized: now 1790219224671888 us; error 41 us; skew 500 ppm
Sep 24 03:07:04 dist-test-slave-wgsb krb5kdc[22381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:07:04.681885 22394 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:07:04.682438 22394 webserver.cc:460] Webserver started at http://127.0.98.62:41545/ using document root <none> and password file <none>
I20260924 03:07:04.682687 22394 fs_manager.cc:359] Metadata directory not provided
I20260924 03:07:04.682736 22394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:07:04.682883 22394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:07:04.684043 22394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "c3c0a8ef80d1425ea842a044d5446a43"
format_stamp: "Formatted at 2026-09-24 03:07:04 on dist-test-slave-wgsb"
server_key: "44d12f714f63f69b76e5e644fcca4d56"
server_key_iv: "68cbefacdfeac245d5b977740689c2bc"
server_key_version: "encryptionkey@0"
I20260924 03:07:04.684402 22394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "c3c0a8ef80d1425ea842a044d5446a43"
format_stamp: "Formatted at 2026-09-24 03:07:04 on dist-test-slave-wgsb"
server_key: "44d12f714f63f69b76e5e644fcca4d56"
server_key_iv: "68cbefacdfeac245d5b977740689c2bc"
server_key_version: "encryptionkey@0"
I20260924 03:07:04.685729 22394 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:07:04.686568 22409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:07:04.686780 22394 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:07:04.686882 22394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "c3c0a8ef80d1425ea842a044d5446a43"
format_stamp: "Formatted at 2026-09-24 03:07:04 on dist-test-slave-wgsb"
server_key: "44d12f714f63f69b76e5e644fcca4d56"
server_key_iv: "68cbefacdfeac245d5b977740689c2bc"
server_key_version: "encryptionkey@0"
I20260924 03:07:04.686985 22394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:07:04.716225 22394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:07:04.718382 22394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:07:04.718534 22394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:07:04.723129 22394 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34341
I20260924 03:07:04.723275 22461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34341 every 8 connection(s)
I20260924 03:07:04.723528 22394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:07:04.724571 22462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:07:04.727133 22462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3c0a8ef80d1425ea842a044d5446a43: Bootstrap starting.
I20260924 03:07:04.727842 22462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3c0a8ef80d1425ea842a044d5446a43: Neither blocks nor log segments found. Creating new log.
I20260924 03:07:04.728154 22462 log.cc:824] T 00000000000000000000000000000000 P c3c0a8ef80d1425ea842a044d5446a43: Log is configured to *not* fsync() on all Append() calls
I20260924 03:07:04.729187 22462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3c0a8ef80d1425ea842a044d5446a43: No bootstrap required, opened a new log
I20260924 03:07:04.730947 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 22394
I20260924 03:07:04.731057 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:07:04.731251 392 external_mini_cluster.cc:1428] Setting key 6efb055b6549dcb15ccfcc6ed6e0677c
I20260924 03:07:04.732332 22462 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3c0a8ef80d1425ea842a044d5446a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c0a8ef80d1425ea842a044d5446a43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34341 } }
I20260924 03:07:04.732528 22462 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3c0a8ef80d1425ea842a044d5446a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:07:04.732575 22462 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3c0a8ef80d1425ea842a044d5446a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3c0a8ef80d1425ea842a044d5446a43, State: Initialized, Role: FOLLOWER
I20260924 03:07:04.732744 22462 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3c0a8ef80d1425ea842a044d5446a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c0a8ef80d1425ea842a044d5446a43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34341 } }
I20260924 03:07:04.732940 22462 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3c0a8ef80d1425ea842a044d5446a43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:07:04.732999 22462 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3c0a8ef80d1425ea842a044d5446a43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:07:04.733057 22462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3c0a8ef80d1425ea842a044d5446a43 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:07:04.734319 22462 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3c0a8ef80d1425ea842a044d5446a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c0a8ef80d1425ea842a044d5446a43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34341 } }
I20260924 03:07:04.734452 22462 leader_election.cc:302] T 00000000000000000000000000000000 P c3c0a8ef80d1425ea842a044d5446a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3c0a8ef80d1425ea842a044d5446a43; no voters:
I20260924 03:07:04.734665 22462 leader_election.cc:288] T 00000000000000000000000000000000 P c3c0a8ef80d1425ea842a044d5446a43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:07:04.734695 22467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3c0a8ef80d1425ea842a044d5446a43 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:07:04.734802 22467 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3c0a8ef80d1425ea842a044d5446a43 [term 1 LEADER]: Becoming Leader. State: Replica: c3c0a8ef80d1425ea842a044d5446a43, State: Running, Role: LEADER
I20260924 03:07:04.734877 22467 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3c0a8ef80d1425ea842a044d5446a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c0a8ef80d1425ea842a044d5446a43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34341 } }
I20260924 03:07:04.734964 22462 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3c0a8ef80d1425ea842a044d5446a43 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:07:04.735200 22467 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3c0a8ef80d1425ea842a044d5446a43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3c0a8ef80d1425ea842a044d5446a43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c0a8ef80d1425ea842a044d5446a43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34341 } } }
I20260924 03:07:04.735258 22467 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3c0a8ef80d1425ea842a044d5446a43 [sys.catalog]: This master's current role is: LEADER
I20260924 03:07:04.735384 22467 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3c0a8ef80d1425ea842a044d5446a43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3c0a8ef80d1425ea842a044d5446a43. Latest consensus state: current_term: 1 leader_uuid: "c3c0a8ef80d1425ea842a044d5446a43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c0a8ef80d1425ea842a044d5446a43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34341 } } }
I20260924 03:07:04.735446 22467 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3c0a8ef80d1425ea842a044d5446a43 [sys.catalog]: This master's current role is: LEADER
W20260924 03:07:04.736824 22482 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c3c0a8ef80d1425ea842a044d5446a43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:07:04.737465 22482 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:07:04.737547 22471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:07:04.737824 22471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:07:04.739728 22471 catalog_manager.cc:1347] Generated new cluster ID: c1b5ca9f6ea140828ba614180c9c7276
I20260924 03:07:04.739859 22471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:07:04.742383 22466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:04.731953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60211 (local address 127.0.98.62:34341)
0924 03:07:04.732802 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:04.732814 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:04.732862 (+ 48us) server_negotiation.cc:407] Connection header received
0924 03:07:04.733011 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:04.733019 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:04.733750 (+ 731us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:04.733902 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:04.734632 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:04.738790 (+ 4158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:04.740074 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:04.740706 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:04.740813 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:04.741614 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:04.741618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:04.741618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:07:04.741854 (+ 236us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:04.741895 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:07:04.741910 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:04.741985 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:04.742075 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 03:07:04.742138 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":109,"threads_started":1}
I20260924 03:07:04.743343 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:07:04.747108 22471 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:07:04.747792 22471 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:07:04.750530 22466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:04.746832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58841 (local address 127.0.98.62:34341)
0924 03:07:04.746873 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:04.746875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:04.747035 (+ 160us) server_negotiation.cc:407] Connection header received
0924 03:07:04.747088 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:04.747089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:04.747102 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:04.747172 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:04.747367 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:04.748160 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:04.749145 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:04.749482 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:04.749552 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:04.750262 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:04.750263 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:04.750263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:07:04.750269 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:04.750286 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:07:04.750290 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:04.750357 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:04.750406 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 03:07:04.750450 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:07:04.756570 22471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3c0a8ef80d1425ea842a044d5446a43: Generated new TSK 0
I20260924 03:07:04.756784 22471 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:07:04.761112 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 22394
2026-09-24T03:07:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:07:04 dist-test-slave-wgsb krb5kdc[22493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:07:04 dist-test-slave-wgsb krb5kdc[22493](info): set up 2 sockets
Sep 24 03:07:04 dist-test-slave-wgsb krb5kdc[22493](info): commencing operation
krb5kdc: starting...
W20260924 03:07:05.835893 392 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:07:05 dist-test-slave-wgsb krb5kdc[22493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:07:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:07:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:07:05.960729 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36085
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:07:06.046124 22506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:07:06.046380 22506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:07:06.046419 22506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:07:06.047616 22506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:07:06.047688 22506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:07:06.047713 22506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:07:06.047744 22506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:07:06.047775 22506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:07:06.049263 22506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:07:06.049537 22506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:07:06.049767 22506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:07:06.054798 22512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:07:06.054792 22514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:07:06.054888 22506 server_base.cc:1034] running on GCE node
W20260924 03:07:06.054792 22511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:07:06.055325 22506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:07:06.055712 22506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:07:06.056936 22506 hybrid_clock.cc:648] HybridClock initialized: now 1790219226056901 us; error 37 us; skew 500 ppm
Sep 24 03:07:06 dist-test-slave-wgsb krb5kdc[22493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:07:06.059866 22506 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:07:06.060308 22506 webserver.cc:460] Webserver started at http://127.0.98.62:34847/ using document root <none> and password file <none>
I20260924 03:07:06.060573 22506 fs_manager.cc:359] Metadata directory not provided
I20260924 03:07:06.060647 22506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:07:06.060801 22506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:07:06.061878 22506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "c9c798965d294b009e9f20323678138a"
format_stamp: "Formatted at 2026-09-24 03:07:06 on dist-test-slave-wgsb"
server_key: "6e4c4cf55fe999ce5a3ac930a96fc687"
server_key_iv: "7ee79dbd286f4c3ece049ea14455d420"
server_key_version: "encryptionkey@0"
I20260924 03:07:06.062250 22506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "c9c798965d294b009e9f20323678138a"
format_stamp: "Formatted at 2026-09-24 03:07:06 on dist-test-slave-wgsb"
server_key: "6e4c4cf55fe999ce5a3ac930a96fc687"
server_key_iv: "7ee79dbd286f4c3ece049ea14455d420"
server_key_version: "encryptionkey@0"
I20260924 03:07:06.063691 22506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:07:06.064484 22521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:07:06.064648 22506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:07:06.064745 22506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "c9c798965d294b009e9f20323678138a"
format_stamp: "Formatted at 2026-09-24 03:07:06 on dist-test-slave-wgsb"
server_key: "6e4c4cf55fe999ce5a3ac930a96fc687"
server_key_iv: "7ee79dbd286f4c3ece049ea14455d420"
server_key_version: "encryptionkey@0"
I20260924 03:07:06.064829 22506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:07:06.068217 22506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:07:06.070168 22506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:07:06.070329 22506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:07:06.074627 22506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36085
I20260924 03:07:06.074671 22573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36085 every 8 connection(s)
I20260924 03:07:06.075033 22506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:07:06.075734 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 22506
I20260924 03:07:06.075862 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:07:06.076072 392 external_mini_cluster.cc:1428] Setting key 446666df75c3b3e47010e31a8345ecad
I20260924 03:07:06.076411 22574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:07:06.079082 22577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:06.076651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58253 (local address 127.0.98.62:36085)
0924 03:07:06.076809 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:06.076816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:06.076838 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:07:06.077651 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:06.077666 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:06.077716 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:06.077883 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:06.078595 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:06.078598 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:06.078599 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:07:06.078603 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:06.078646 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:07:06.078660 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:06.078725 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:06.078838 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 03:07:06.078878 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
I20260924 03:07:06.080034 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:07:06.080470 22574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9c798965d294b009e9f20323678138a: Bootstrap starting.
I20260924 03:07:06.081159 22574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9c798965d294b009e9f20323678138a: Neither blocks nor log segments found. Creating new log.
I20260924 03:07:06.081499 22574 log.cc:824] T 00000000000000000000000000000000 P c9c798965d294b009e9f20323678138a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:07:06.082491 22577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:06.081503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38439 (local address 127.0.98.62:36085)
0924 03:07:06.081535 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:06.081537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:06.081647 (+ 110us) server_negotiation.cc:407] Connection header received
0924 03:07:06.081722 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:06.081723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:06.081728 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:06.081791 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:06.082324 (+ 533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:06.082325 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:06.082325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:07:06.082328 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:06.082340 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:07:06.082343 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:06.082393 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:06.082441 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 03:07:06.082451 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:07:06.082538 22574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9c798965d294b009e9f20323678138a: No bootstrap required, opened a new log
I20260924 03:07:06.084431 22574 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9c798965d294b009e9f20323678138a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c798965d294b009e9f20323678138a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36085 } }
I20260924 03:07:06.084563 22574 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9c798965d294b009e9f20323678138a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:07:06.084586 22574 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9c798965d294b009e9f20323678138a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9c798965d294b009e9f20323678138a, State: Initialized, Role: FOLLOWER
I20260924 03:07:06.084697 22574 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9c798965d294b009e9f20323678138a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c798965d294b009e9f20323678138a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36085 } }
I20260924 03:07:06.084787 22574 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9c798965d294b009e9f20323678138a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:07:06.084810 22574 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9c798965d294b009e9f20323678138a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:07:06.084837 22574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9c798965d294b009e9f20323678138a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:07:06.085307 22574 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9c798965d294b009e9f20323678138a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c798965d294b009e9f20323678138a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36085 } }
I20260924 03:07:06.085388 22574 leader_election.cc:302] T 00000000000000000000000000000000 P c9c798965d294b009e9f20323678138a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9c798965d294b009e9f20323678138a; no voters:
I20260924 03:07:06.085511 22574 leader_election.cc:288] T 00000000000000000000000000000000 P c9c798965d294b009e9f20323678138a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:07:06.085579 22584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9c798965d294b009e9f20323678138a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:07:06.085755 22574 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9c798965d294b009e9f20323678138a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:07:06.085783 22584 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9c798965d294b009e9f20323678138a [term 1 LEADER]: Becoming Leader. State: Replica: c9c798965d294b009e9f20323678138a, State: Running, Role: LEADER
I20260924 03:07:06.085881 22584 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9c798965d294b009e9f20323678138a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c798965d294b009e9f20323678138a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36085 } }
I20260924 03:07:06.086272 22586 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9c798965d294b009e9f20323678138a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9c798965d294b009e9f20323678138a. Latest consensus state: current_term: 1 leader_uuid: "c9c798965d294b009e9f20323678138a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c798965d294b009e9f20323678138a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36085 } } }
I20260924 03:07:06.086400 22586 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9c798965d294b009e9f20323678138a [sys.catalog]: This master's current role is: LEADER
I20260924 03:07:06.086262 22585 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9c798965d294b009e9f20323678138a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9c798965d294b009e9f20323678138a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c798965d294b009e9f20323678138a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36085 } } }
I20260924 03:07:06.086607 22593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:07:06.086604 22585 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9c798965d294b009e9f20323678138a [sys.catalog]: This master's current role is: LEADER
I20260924 03:07:06.087131 22593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:07:06.088753 22593 catalog_manager.cc:1347] Generated new cluster ID: 583eb5c3b79a4db3bc8fee7c8ef68c63
I20260924 03:07:06.088850 22593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:07:06.095829 22593 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:07:06.096001 22593 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:07:06.098186 22593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9c798965d294b009e9f20323678138a: Generated new TSK 0
I20260924 03:07:06.098415 22593 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:07:06.102687 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 22506
2026-09-24T03:07:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1338 ms)
[----------] 18 tests from AuthTokenIssuingTest (25603 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:07:06 dist-test-slave-wgsb krb5kdc[22605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:07:06 dist-test-slave-wgsb krb5kdc[22605](info): set up 2 sockets
Sep 24 03:07:06 dist-test-slave-wgsb krb5kdc[22605](info): commencing operation
krb5kdc: starting...
W20260924 03:07:07.183858 392 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:07:07 dist-test-slave-wgsb krb5kdc[22605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:07:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:07:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:07:07.310951 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36047
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36047,127.0.98.61:40225,127.0.98.60:45785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:07:07.395555 22618 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:07:07.395835 22618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:07:07.395854 22618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:07:07.395876 22618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:07:07.397174 22618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:07:07.397236 22618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:07:07.397253 22618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:07:07.397265 22618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:07:07.397284 22618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:07:07.398898 22618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36047,127.0.98.61:40225,127.0.98.60:45785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:07:07.399178 22618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:07:07.399430 22618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:07:07.404481 22623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:07:07.404522 22624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:07:07.404564 22626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:07:07.404909 22618 server_base.cc:1034] running on GCE node
I20260924 03:07:07.405138 22618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:07:07.405381 22618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:07:07.406534 22618 hybrid_clock.cc:648] HybridClock initialized: now 1790219227406519 us; error 30 us; skew 500 ppm
Sep 24 03:07:07 dist-test-slave-wgsb krb5kdc[22605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:07:07.409107 22618 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:07:07.409566 22618 webserver.cc:460] Webserver started at http://127.0.98.62:43835/ using document root <none> and password file <none>
I20260924 03:07:07.409765 22618 fs_manager.cc:359] Metadata directory not provided
I20260924 03:07:07.409814 22618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:07:07.409904 22618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:07:07.410970 22618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "dffc328662d444589521930b74d45d4c"
format_stamp: "Formatted at 2026-09-24 03:07:07 on dist-test-slave-wgsb"
server_key: "e0e58829e4d3b7e33094eb0e4be41ccf"
server_key_iv: "12e2ad913abbceb179baac821ecf9cb5"
server_key_version: "encryptionkey@0"
I20260924 03:07:07.411278 22618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "dffc328662d444589521930b74d45d4c"
format_stamp: "Formatted at 2026-09-24 03:07:07 on dist-test-slave-wgsb"
server_key: "e0e58829e4d3b7e33094eb0e4be41ccf"
server_key_iv: "12e2ad913abbceb179baac821ecf9cb5"
server_key_version: "encryptionkey@0"
I20260924 03:07:07.412796 22618 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:07:07.413743 22633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:07:07.414245 22618 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:07:07.414884 22618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "dffc328662d444589521930b74d45d4c"
format_stamp: "Formatted at 2026-09-24 03:07:07 on dist-test-slave-wgsb"
server_key: "e0e58829e4d3b7e33094eb0e4be41ccf"
server_key_iv: "12e2ad913abbceb179baac821ecf9cb5"
server_key_version: "encryptionkey@0"
I20260924 03:07:07.415014 22618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:07:07.440415 22618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:07:07.442855 22618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:07:07.443048 22618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:07:07.447227 22618 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36047
I20260924 03:07:07.447269 22685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36047 every 8 connection(s)
I20260924 03:07:07.447707 22618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:07:07.448740 22686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:07:07.450366 22686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36047 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:07:07.456513 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 22618
I20260924 03:07:07.456622 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:07:07.456812 392 external_mini_cluster.cc:1428] Setting key cacfa203cef99dc91abec12461ce36e5
Sep 24 03:07:07 dist-test-slave-wgsb krb5kdc[22605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:07:07.461469 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.451022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36047 (local address 127.0.0.1:40108)
0924 03:07:07.451228 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:07.451331 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0924 03:07:07.451905 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:07:07.452256 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:07:07.452259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:07:07.452519 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:07:07.452668 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:07.452672 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.453791 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.453792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:07.454457 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:07.454458 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.454981 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.454982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:07.455038 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:07.455663 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:07:07.455670 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:07:07.457550 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:07:07.459897 (+ 2347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:07.459904 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:07.459905 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:07.460104 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:07.460392 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:07.460393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:07.460393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:07.460473 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:07.460976 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:07:07.460982 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:07:07.461050 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 03:07:07.461154 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 03:07:07.461309 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":82,"threads_started":1}
I20260924 03:07:07.461627 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.451194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40108 (local address 127.0.98.62:36047)
0924 03:07:07.451348 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:07.451354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:07.451418 (+ 64us) server_negotiation.cc:407] Connection header received
0924 03:07:07.452093 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:07.452095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:07.452114 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:07.452212 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:07.452795 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.453698 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.454536 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.454889 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.454982 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:07.457659 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:07.457667 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:07.457668 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:07.457675 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:07.459803 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.460191 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.460195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.460195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.460318 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.460544 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.460545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.460545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.460873 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:07.460978 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:07.461202 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 03:07:07.461419 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":120,"thread_start_us":158,"threads_started":1}
I20260924 03:07:07.462371 22686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40225 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:07:07.462754 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.462614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40225 (local address 127.0.0.1:49160)
0924 03:07:07.462660 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:07.462707 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 03:07:07.462930 22636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40225: connect: Connection refused (error 111)
I20260924 03:07:07.463399 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.463300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40225 (local address 127.0.0.1:49164)
0924 03:07:07.463348 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:07.463372 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 03:07:07.463533 22686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40225: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40225: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 03:07:07.486966 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40225
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36047,127.0.98.61:40225,127.0.98.60:45785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:07:07.512809 22686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40225 } attempt: 1
I20260924 03:07:07.513371 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.513181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40225 (local address 127.0.0.1:49166)
0924 03:07:07.513241 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:07.513341 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 03:07:07.513744 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.513649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40225 (local address 127.0.0.1:49176)
0924 03:07:07.513696 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:07.513722 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 03:07:07.513861 22686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40225: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40225: connect: Connection refused (error 111)
W20260924 03:07:07.569041 22693 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:07:07.569339 22693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:07:07.569384 22693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:07:07.569433 22693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:07:07.570619 22693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:07:07.570693 22693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:07:07.570716 22693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:07:07.570736 22693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:07:07.570770 22693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:07:07.572386 22693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36047,127.0.98.61:40225,127.0.98.60:45785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:07:07.572680 22693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:07:07.572924 22693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:07:07.580924 22698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:07:07.581166 22693 server_base.cc:1034] running on GCE node
I20260924 03:07:07.582042 22686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40225 } attempt: 2
I20260924 03:07:07.583086 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.582738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40225 (local address 127.0.0.1:49188)
0924 03:07:07.583005 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:07.583043 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234}
W20260924 03:07:07.583086 22699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:07:07.583348 22701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:07:07.583706 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.583560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40225 (local address 127.0.0.1:49194)
0924 03:07:07.583637 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:07.583672 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 03:07:07.583760 22693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20260924 03:07:07.583871 22686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40225: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40225: connect: Connection refused (error 111)
I20260924 03:07:07.584138 22693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:07:07.585421 22693 hybrid_clock.cc:648] HybridClock initialized: now 1790219227585370 us; error 60 us; skew 500 ppm
Sep 24 03:07:07 dist-test-slave-wgsb krb5kdc[22605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:07:07.590976 22693 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 03:07:07.591672 22693 webserver.cc:460] Webserver started at http://127.0.98.61:43403/ using document root <none> and password file <none>
I20260924 03:07:07.591881 22693 fs_manager.cc:359] Metadata directory not provided
I20260924 03:07:07.591925 22693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:07:07.592027 22693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:07:07.593227 22693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-1/data/instance:
uuid: "996dd202e939431683c274b79bddefde"
format_stamp: "Formatted at 2026-09-24 03:07:07 on dist-test-slave-wgsb"
server_key: "401478ef0185efb9b56380b18616166f"
server_key_iv: "18fac19752ca617254b046eeda53e424"
server_key_version: "encryptionkey@0"
I20260924 03:07:07.593550 22693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-1/wal/instance:
uuid: "996dd202e939431683c274b79bddefde"
format_stamp: "Formatted at 2026-09-24 03:07:07 on dist-test-slave-wgsb"
server_key: "401478ef0185efb9b56380b18616166f"
server_key_iv: "18fac19752ca617254b046eeda53e424"
server_key_version: "encryptionkey@0"
I20260924 03:07:07.594969 22693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:07:07.595908 22708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:07:07.596190 22693 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:07:07.596313 22693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-1/wal
uuid: "996dd202e939431683c274b79bddefde"
format_stamp: "Formatted at 2026-09-24 03:07:07 on dist-test-slave-wgsb"
server_key: "401478ef0185efb9b56380b18616166f"
server_key_iv: "18fac19752ca617254b046eeda53e424"
server_key_version: "encryptionkey@0"
I20260924 03:07:07.596381 22693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:07:07.610023 22693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:07:07.612202 22693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:07:07.612442 22693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:07:07.616752 22693 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40225
I20260924 03:07:07.616835 22760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40225 every 8 connection(s)
I20260924 03:07:07.617173 22693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-1/data/info.pb
I20260924 03:07:07.618297 22761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:07:07.619966 22761 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36047 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:07:07.622750 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 22693
I20260924 03:07:07.622864 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-1/wal/instance
I20260924 03:07:07.623029 392 external_mini_cluster.cc:1428] Setting key 6a3e52c52bafc5939f49aa9bac3c3c45
Sep 24 03:07:07 dist-test-slave-wgsb krb5kdc[22605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:07:07.630326 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.620747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40114 (local address 127.0.98.62:36047)
0924 03:07:07.620857 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:07.620859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:07.621046 (+ 187us) server_negotiation.cc:407] Connection header received
0924 03:07:07.621570 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:07.621572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:07.621588 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:07.621658 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:07.622410 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.623147 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.624301 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.624517 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.624595 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:07.627234 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:07.627236 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:07.627237 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:07.627245 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:07.628887 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.629335 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.629336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.629336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.629399 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.629660 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.629661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.629661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.629784 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:07.629862 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:07.630141 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 03:07:07.630229 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 03:07:07.630542 22763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.620665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36047 (local address 127.0.0.1:40114)
0924 03:07:07.620913 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:07.620959 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 03:07:07.621462 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:07:07.621755 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:07:07.621758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:07:07.622057 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:07:07.622250 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:07.622254 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.623273 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.623275 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:07.624179 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:07.624182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.624632 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.624633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:07.624755 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:07.625379 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:07:07.625388 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:07:07.627126 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:07:07.628998 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:07.629006 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:07.629009 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:07.629243 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:07.629474 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:07.629475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:07.629475 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:07.629587 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:07.629861 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:07:07.629868 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:07:07.630013 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 03:07:07.630121 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 03:07:07.630282 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260924 03:07:07.631127 22761 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40225 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 03:07:07 dist-test-slave-wgsb krb5kdc[22605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:07:07.638811 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.631489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49206 (local address 127.0.98.61:40225)
0924 03:07:07.631683 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:07.631688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:07.631713 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:07:07.631735 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:07.631737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:07.631748 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:07.631811 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:07.632138 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.632922 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.633584 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.633963 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.634054 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:07.635930 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:07.635937 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:07.635937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:07.635943 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:07.637543 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.637965 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.637968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.637969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.638034 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.638286 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.638286 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.638287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.638420 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:07.638486 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:07.638652 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 03:07:07.638720 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":131,"threads_started":1}
I20260924 03:07:07.639041 22763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.631451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40225 (local address 127.0.0.1:49206)
0924 03:07:07.631501 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:07.631511 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:07:07.631547 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:07:07.631870 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:07:07.631871 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:07:07.631985 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:07:07.632037 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:07.632038 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.633008 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.633009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:07.633504 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:07.633505 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.634092 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.634093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:07.634165 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:07.634528 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:07:07.634534 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:07:07.635831 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:07:07.637632 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:07.637634 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:07.637634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:07.637879 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:07.638140 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:07.638142 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:07.638142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:07.638217 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:07.638492 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:07:07.638493 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:07:07.638546 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 03:07:07.638630 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 03:07:07.638799 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":47}
I20260924 03:07:07.639868 22761 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45785 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:07:07.640244 22763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.640101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45785 (local address 127.0.0.1:37830)
0924 03:07:07.640157 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:07.640211 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 03:07:07.640425 22710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45785: connect: Connection refused (error 111)
I20260924 03:07:07.640859 22763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.640756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45785 (local address 127.0.0.1:37844)
0924 03:07:07.640803 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:07.640833 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 03:07:07.641021 22761 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45785: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45785: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 03:07:07.655301 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45785
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36047,127.0.98.61:40225,127.0.98.60:45785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:07:07.675047 22686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40225 } attempt: 3
Sep 24 03:07:07 dist-test-slave-wgsb krb5kdc[22605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:07:07.683214 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.675464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40225 (local address 127.0.0.1:49212)
0924 03:07:07.675517 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:07.675534 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:07:07.675595 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:07:07.675798 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:07:07.675800 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:07:07.675958 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:07:07.676023 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:07.676025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.676818 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.676819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:07.677338 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:07.677339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.677769 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.677770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:07.677845 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:07.678398 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:07:07.678402 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:07:07.679790 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:07:07.681473 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:07.681474 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:07.681474 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:07.681703 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:07.681950 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:07.681951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:07.681951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:07.682021 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:07.682643 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:07:07.682645 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:07:07.682896 (+ 251us) client_negotiation.cc:769] Sending connection context
0924 03:07:07.682996 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 03:07:07.683107 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 03:07:07.683285 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.675536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49212 (local address 127.0.98.61:40225)
0924 03:07:07.675636 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:07.675639 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:07.675649 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:07:07.675679 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:07.675679 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:07.675693 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:07.675763 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:07.676111 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.676709 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.677426 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.677648 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.677751 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:07.679893 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:07.679894 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:07.679895 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:07.679902 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:07.681380 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.681798 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.681799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.681800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.681864 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.682105 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.682106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.682107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.682361 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:07.682430 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:07.683041 (+ 611us) server_negotiation.cc:299] Negotiation successful
0924 03:07:07.683162 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 03:07:07.683800 22686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45785 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:07:07.684199 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.684085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45785 (local address 127.0.0.1:37854)
0924 03:07:07.684135 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:07.684174 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 03:07:07.684545 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.684452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45785 (local address 127.0.0.1:37858)
0924 03:07:07.684500 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:07.684527 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 03:07:07.684638 22686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45785: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45785: connect: Connection refused (error 111)
I20260924 03:07:07.690287 22761 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45785 } attempt: 1
I20260924 03:07:07.690789 22763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.690637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45785 (local address 127.0.0.1:37872)
0924 03:07:07.690703 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:07.690748 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 03:07:07.691171 22763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.691075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45785 (local address 127.0.0.1:37882)
0924 03:07:07.691124 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:07.691153 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 03:07:07.691323 22761 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45785: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45785: connect: Connection refused (error 111)
I20260924 03:07:07.715809 22686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45785 } attempt: 1
I20260924 03:07:07.716466 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.716298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45785 (local address 127.0.0.1:37888)
0924 03:07:07.716369 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:07.716421 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 03:07:07.716899 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.716793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45785 (local address 127.0.0.1:37890)
0924 03:07:07.716845 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:07.716874 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 03:07:07.717015 22686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45785: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45785: connect: Connection refused (error 111)
W20260924 03:07:07.744585 22768 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:07:07.744818 22768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:07:07.744836 22768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:07:07.744858 22768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:07:07.746083 22768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:07:07.746136 22768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:07:07.746150 22768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:07:07.746163 22768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:07:07.746176 22768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:07:07.747830 22768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36047,127.0.98.61:40225,127.0.98.60:45785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:07:07.748112 22768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:07:07.748368 22768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:07:07.753394 22776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:07:07.753398 22773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:07:07.753396 22774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:07:07.753804 22768 server_base.cc:1034] running on GCE node
I20260924 03:07:07.754050 22768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:07:07.754348 22768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:07:07.755545 22768 hybrid_clock.cc:648] HybridClock initialized: now 1790219227755526 us; error 52 us; skew 500 ppm
Sep 24 03:07:07 dist-test-slave-wgsb krb5kdc[22605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:07:07.758562 22768 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 03:07:07.759162 22768 webserver.cc:460] Webserver started at http://127.0.98.60:40845/ using document root <none> and password file <none>
I20260924 03:07:07.759482 22768 fs_manager.cc:359] Metadata directory not provided
I20260924 03:07:07.759572 22768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:07:07.759507 22761 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45785 } attempt: 2
I20260924 03:07:07.759819 22768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:07:07.760067 22763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.759927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45785 (local address 127.0.0.1:37894)
0924 03:07:07.759984 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:07.760019 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 03:07:07.760571 22763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.760307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45785 (local address 127.0.0.1:37896)
0924 03:07:07.760429 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:07.760495 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260924 03:07:07.760778 22761 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45785: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45785: connect: Connection refused (error 111)
I20260924 03:07:07.761330 22768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-2/data/instance:
uuid: "f949abb18bff4316a7b9427409201be2"
format_stamp: "Formatted at 2026-09-24 03:07:07 on dist-test-slave-wgsb"
server_key: "8c552b20a5e512ef94dfd25e6845ce83"
server_key_iv: "1f0c8052f098fe8509b018e2b8b4b88c"
server_key_version: "encryptionkey@0"
I20260924 03:07:07.761754 22768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-2/wal/instance:
uuid: "f949abb18bff4316a7b9427409201be2"
format_stamp: "Formatted at 2026-09-24 03:07:07 on dist-test-slave-wgsb"
server_key: "8c552b20a5e512ef94dfd25e6845ce83"
server_key_iv: "1f0c8052f098fe8509b018e2b8b4b88c"
server_key_version: "encryptionkey@0"
I20260924 03:07:07.763257 22768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260924 03:07:07.764194 22783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:07:07.764397 22768 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 03:07:07.764489 22768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-2/wal
uuid: "f949abb18bff4316a7b9427409201be2"
format_stamp: "Formatted at 2026-09-24 03:07:07 on dist-test-slave-wgsb"
server_key: "8c552b20a5e512ef94dfd25e6845ce83"
server_key_iv: "1f0c8052f098fe8509b018e2b8b4b88c"
server_key_version: "encryptionkey@0"
I20260924 03:07:07.764549 22768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:07:07.777587 22768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:07:07.780347 22768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:07:07.780472 22768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:07:07.784438 22768 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45785
I20260924 03:07:07.784518 22835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45785 every 8 connection(s)
I20260924 03:07:07.784835 22768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-2/data/info.pb
I20260924 03:07:07.786032 22836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:07:07.787891 22836 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36047 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:07:07.790647 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 22768
I20260924 03:07:07.790778 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218948286999-392-0/minicluster-data/master-2/wal/instance
I20260924 03:07:07.790984 392 external_mini_cluster.cc:1428] Setting key a67f010a8fcf38c5bef5f874426fe4a9
I20260924 03:07:07.792146 22686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45785 } attempt: 2
Sep 24 03:07:07 dist-test-slave-wgsb krb5kdc[22605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 03:07:07 dist-test-slave-wgsb krb5kdc[22605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 03:07:07 dist-test-slave-wgsb krb5kdc[22605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 03:07:07.800598 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.788965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40124 (local address 127.0.98.62:36047)
0924 03:07:07.789134 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:07.789138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:07.789397 (+ 259us) server_negotiation.cc:407] Connection header received
0924 03:07:07.789945 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:07.789947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:07.789960 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:07.790015 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:07.790684 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.791315 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.793032 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.793268 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.793358 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:07.796660 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:07.796662 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:07.796663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:07.796670 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:07.798624 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.799228 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.799230 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.799230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.799307 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.799664 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.799666 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.799666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.799888 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:07.799985 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:07.800416 (+ 431us) server_negotiation.cc:299] Negotiation successful
0924 03:07:07.800498 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260924 03:07:07.801015 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.792389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45785 (local address 127.0.0.1:37914)
0924 03:07:07.792437 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:07.792451 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:07:07.792479 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:07:07.792723 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:07:07.792724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:07:07.792881 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:07:07.792916 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:07.792918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.793802 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.793803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:07.794318 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:07.794319 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.794905 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.794907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:07.794974 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:07.795474 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:07:07.795480 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:07:07.797108 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:07:07.799482 (+ 2374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:07.799483 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:07.799483 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:07.799725 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:07.800276 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:07.800277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:07.800277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:07.800338 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:07.800740 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:07:07.800741 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:07:07.800794 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 03:07:07.800840 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 03:07:07.800925 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 03:07:07.801245 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.788889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36047 (local address 127.0.0.1:40124)
0924 03:07:07.789206 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:07.789251 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:07:07.789820 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:07:07.790106 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:07:07.790109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:07:07.790369 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:07:07.790590 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:07.790594 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.791415 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.791416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:07.792935 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:07.792938 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.793375 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.793376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:07.793505 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:07.794057 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:07:07.794068 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:07:07.796555 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:07:07.798707 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:07.798714 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:07.798716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:07.799106 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:07.799408 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:07.799409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:07.799409 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:07.799558 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:07.800043 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:07:07.800050 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:07:07.800247 (+ 197us) client_negotiation.cc:769] Sending connection context
0924 03:07:07.800377 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 03:07:07.800567 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":153,"threads_started":1}
I20260924 03:07:07.801493 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.792444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37914 (local address 127.0.98.60:45785)
0924 03:07:07.792595 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:07.792597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:07.792603 (+ 6us) server_negotiation.cc:407] Connection header received
0924 03:07:07.792621 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:07.792621 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:07.792631 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:07.792700 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:07.792996 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.793680 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.794409 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.794706 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.794832 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:07.797231 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:07.797238 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:07.797239 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:07.797261 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:07.799359 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.800104 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.800108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.800108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.800217 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.800413 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.800414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.800415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.800586 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:07.801249 (+ 663us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:07.801299 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 03:07:07.801381 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260924 03:07:07.801985 22836 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40225 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:07:07.802737 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.791433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37910 (local address 127.0.98.60:45785)
0924 03:07:07.791678 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:07.791683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:07.791727 (+ 44us) server_negotiation.cc:407] Connection header received
0924 03:07:07.791746 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:07.791748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:07.791764 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:07.791823 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:07.792572 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.793624 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.795062 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.795385 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.795455 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:07.798674 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:07.798676 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:07.798677 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:07.798684 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:07.800876 (+ 2192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.801344 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.801346 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.801346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.801416 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.801660 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.801661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.801661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.801971 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:07.802104 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:07.802479 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 03:07:07.802591 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":110,"threads_started":1}
I20260924 03:07:07.803279 22686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c: Bootstrap starting.
I20260924 03:07:07.804275 22686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c: Neither blocks nor log segments found. Creating new log.
I20260924 03:07:07.804590 22686 log.cc:824] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:07:07.805838 22686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c: No bootstrap required, opened a new log
Sep 24 03:07:07 dist-test-slave-wgsb krb5kdc[22605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:07:07.807934 22686 raft_consensus.cc:348] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dffc328662d444589521930b74d45d4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36047 } } peers { permanent_uuid: "996dd202e939431683c274b79bddefde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40225 } } peers { permanent_uuid: "f949abb18bff4316a7b9427409201be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45785 } }
I20260924 03:07:07.808127 22686 raft_consensus.cc:729] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dffc328662d444589521930b74d45d4c, State: Initialized, Role: FOLLOWER
I20260924 03:07:07.808245 22686 consensus_queue.cc:260] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dffc328662d444589521930b74d45d4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36047 } } peers { permanent_uuid: "996dd202e939431683c274b79bddefde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40225 } } peers { permanent_uuid: "f949abb18bff4316a7b9427409201be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45785 } }
Sep 24 03:07:07 dist-test-slave-wgsb krb5kdc[22605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:07:07.808594 22686 sys_catalog.cc:564] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:07:07.809185 22843 sys_catalog.cc:455] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dffc328662d444589521930b74d45d4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36047 } } peers { permanent_uuid: "996dd202e939431683c274b79bddefde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40225 } } peers { permanent_uuid: "f949abb18bff4316a7b9427409201be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45785 } } }
I20260924 03:07:07.809442 22843 sys_catalog.cc:458] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:07:07.810443 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.802276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40225 (local address 127.0.0.1:49226)
0924 03:07:07.802321 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:07.802333 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:07:07.802382 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:07:07.802520 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:07:07.802521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:07:07.802695 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:07:07.802767 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:07.802769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.803655 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.803657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:07.804581 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:07.804583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.805077 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.805078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:07.805158 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:07.805606 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:07:07.805611 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:07:07.807021 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:07:07.809101 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:07.809103 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:07.809103 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:07.809341 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:07.809586 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:07.809587 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:07.809588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:07.809655 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:07.810003 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:07:07.810004 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:07:07.810103 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 03:07:07.810187 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 03:07:07.810307 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 03:07:07.810452 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.802324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49226 (local address 127.0.98.61:40225)
0924 03:07:07.802371 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:07.802373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:07.802383 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:07:07.802441 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:07.802442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:07.802454 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:07.802516 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:07.802864 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.803562 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.804668 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.804973 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.805067 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:07.807123 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:07.807125 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:07.807125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:07.807134 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:07.809010 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.809434 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.809435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.809435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.809500 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.809730 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.809731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.809732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.809891 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:07.809969 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:07.810238 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 03:07:07.810344 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 03:07:07.811323 22836 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45785 } has no permanent_uuid. Determining permanent_uuid...
W20260924 03:07:07.812150 22854 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 03:07:07.812301 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.803507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40130 (local address 127.0.98.62:36047)
0924 03:07:07.803548 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:07.803551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:07.803559 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:07:07.803656 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:07.803657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:07.803667 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:07.803703 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:07.804384 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.805078 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.805832 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.806113 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.806198 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:07.809163 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:07.809165 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:07.809165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:07.809172 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:07.810976 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.811391 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.811392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.811392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.811450 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.811706 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.811707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.811707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.811833 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:07.811998 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:07.812071 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 03:07:07.812147 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 03:07:07.812347 22854 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 24 03:07:07 dist-test-slave-wgsb krb5kdc[22605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 03:07:07 dist-test-slave-wgsb krb5kdc[22605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:07:07.814004 22618 master_runner.cc:186] Error getting master registration for 127.0.98.61:40225: OK, instance_id { permanent_uuid: "996dd202e939431683c274b79bddefde" instance_seqno: 1790219227615484 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:07:07.818377 22618 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:07:07.820528 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.811567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45785 (local address 127.0.0.1:37916)
0924 03:07:07.811690 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:07.811702 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:07:07.811745 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:07:07.812096 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:07:07.812097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:07:07.812269 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:07:07.812309 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:07.812311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.813288 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.813290 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:07.814065 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:07.814067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.815031 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.815032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:07.815080 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:07.815640 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:07:07.815645 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:07:07.817272 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:07:07.819307 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:07.819308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:07.819308 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:07.819487 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:07.819700 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:07.819700 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:07.819701 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:07.819758 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:07.820138 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:07:07.820139 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:07:07.820190 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 03:07:07.820269 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 03:07:07.820365 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260924 03:07:07.820546 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.811677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37916 (local address 127.0.98.60:45785)
0924 03:07:07.811965 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:07.811968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:07.811975 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:07:07.811994 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:07.811995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:07.812004 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:07.812057 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:07.812483 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.813191 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.814144 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.814895 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.814979 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:07.817365 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:07.817366 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:07.817367 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:07.817377 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:07.819228 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.819576 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.819577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.819577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.819651 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.819836 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.819837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.819837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.820050 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:07.820129 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:07.820329 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 03:07:07.820425 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":11,"server-negotiator.queue_time_us":266}
I20260924 03:07:07.821907 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.812833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49230 (local address 127.0.98.61:40225)
0924 03:07:07.812868 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:07.812870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:07.812877 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:07:07.812945 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:07.812946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:07.812954 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:07.812998 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:07.813522 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.814220 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.815553 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.815867 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.816170 (+ 303us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:07.818750 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:07.818751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:07.818751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:07.818759 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:07.820649 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.821055 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.821056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.821056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.821208 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.821404 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.821406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.821406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.821523 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:07.821687 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:07.821725 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 03:07:07.821800 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:07:07.822511 22836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f949abb18bff4316a7b9427409201be2: Bootstrap starting.
I20260924 03:07:07.823122 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:07:07.823289 22836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f949abb18bff4316a7b9427409201be2: Neither blocks nor log segments found. Creating new log.
I20260924 03:07:07.823624 22836 log.cc:824] T 00000000000000000000000000000000 P f949abb18bff4316a7b9427409201be2: Log is configured to *not* fsync() on all Append() calls
I20260924 03:07:07.824497 22836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f949abb18bff4316a7b9427409201be2: No bootstrap required, opened a new log
I20260924 03:07:07.824707 22660 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "dffc328662d444589521930b74d45d4c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40130
I20260924 03:07:07.824865 22660 raft_consensus.cc:482] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 03:07:07.824903 22660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:07:07.825685 22660 raft_consensus.cc:504] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dffc328662d444589521930b74d45d4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36047 } } peers { permanent_uuid: "996dd202e939431683c274b79bddefde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40225 } } peers { permanent_uuid: "f949abb18bff4316a7b9427409201be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45785 } }
I20260924 03:07:07.826011 22660 leader_election.cc:288] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c [CANDIDATE]: Term 1 election: Requested vote from peers 996dd202e939431683c274b79bddefde (127.0.98.61:40225), f949abb18bff4316a7b9427409201be2 (127.0.98.60:45785)
I20260924 03:07:07.826658 22836 raft_consensus.cc:348] T 00000000000000000000000000000000 P f949abb18bff4316a7b9427409201be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dffc328662d444589521930b74d45d4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36047 } } peers { permanent_uuid: "996dd202e939431683c274b79bddefde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40225 } } peers { permanent_uuid: "f949abb18bff4316a7b9427409201be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45785 } }
I20260924 03:07:07.826808 22836 raft_consensus.cc:729] T 00000000000000000000000000000000 P f949abb18bff4316a7b9427409201be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f949abb18bff4316a7b9427409201be2, State: Initialized, Role: FOLLOWER
I20260924 03:07:07.826908 22836 consensus_queue.cc:260] T 00000000000000000000000000000000 P f949abb18bff4316a7b9427409201be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dffc328662d444589521930b74d45d4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36047 } } peers { permanent_uuid: "996dd202e939431683c274b79bddefde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40225 } } peers { permanent_uuid: "f949abb18bff4316a7b9427409201be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45785 } }
I20260924 03:07:07.827221 22836 sys_catalog.cc:564] T 00000000000000000000000000000000 P f949abb18bff4316a7b9427409201be2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:07:07.827291 22857 sys_catalog.cc:455] T 00000000000000000000000000000000 P f949abb18bff4316a7b9427409201be2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dffc328662d444589521930b74d45d4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36047 } } peers { permanent_uuid: "996dd202e939431683c274b79bddefde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40225 } } peers { permanent_uuid: "f949abb18bff4316a7b9427409201be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45785 } } }
I20260924 03:07:07.827478 22857 sys_catalog.cc:458] T 00000000000000000000000000000000 P f949abb18bff4316a7b9427409201be2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:07:07.827733 22810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dffc328662d444589521930b74d45d4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f949abb18bff4316a7b9427409201be2"
I20260924 03:07:07.827860 22810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f949abb18bff4316a7b9427409201be2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:07:07.829018 22810 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f949abb18bff4316a7b9427409201be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dffc328662d444589521930b74d45d4c in term 1.
I20260924 03:07:07.829392 22635 leader_election.cc:302] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dffc328662d444589521930b74d45d4c, f949abb18bff4316a7b9427409201be2; no voters:
I20260924 03:07:07.829572 22843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:07:07.829767 22843 raft_consensus.cc:686] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c [term 1 LEADER]: Becoming Leader. State: Replica: dffc328662d444589521930b74d45d4c, State: Running, Role: LEADER
I20260924 03:07:07.829861 22843 consensus_queue.cc:237] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dffc328662d444589521930b74d45d4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36047 } } peers { permanent_uuid: "996dd202e939431683c274b79bddefde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40225 } } peers { permanent_uuid: "f949abb18bff4316a7b9427409201be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45785 } }
I20260924 03:07:07.830344 22843 sys_catalog.cc:455] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader dffc328662d444589521930b74d45d4c. Latest consensus state: current_term: 1 leader_uuid: "dffc328662d444589521930b74d45d4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dffc328662d444589521930b74d45d4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36047 } } peers { permanent_uuid: "996dd202e939431683c274b79bddefde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40225 } } peers { permanent_uuid: "f949abb18bff4316a7b9427409201be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45785 } } }
I20260924 03:07:07.830426 22843 sys_catalog.cc:458] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c [sys.catalog]: This master's current role is: LEADER
I20260924 03:07:07.831148 22875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:07:07.831440 22875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260924 03:07:07.831539 22868 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f949abb18bff4316a7b9427409201be2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:07:07.831658 22868 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:07:07.832927 22768 master_runner.cc:186] Error getting master registration for 127.0.98.61:40225: OK, instance_id { permanent_uuid: "996dd202e939431683c274b79bddefde" instance_seqno: 1790219227615484 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:07:07.834506 22768 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:07:07.849318 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.840032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40142 (local address 127.0.98.62:36047)
0924 03:07:07.840092 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:07.840094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:07.840415 (+ 321us) server_negotiation.cc:407] Connection header received
0924 03:07:07.840534 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:07.840535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:07.840547 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:07.840579 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:07.841487 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.841963 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.843002 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.843193 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.843313 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:07.845456 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:07.845458 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:07.845459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:07.845468 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:07.847307 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.848057 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.848059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.848060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.848126 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.848461 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.848462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.848463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.848611 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:07.848687 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:07.849097 (+ 410us) server_negotiation.cc:299] Negotiation successful
0924 03:07:07.849188 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:07:07.851831 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.840271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37928 (local address 127.0.98.60:45785)
0924 03:07:07.840330 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:07.840332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:07.843792 (+ 3460us) server_negotiation.cc:407] Connection header received
0924 03:07:07.843851 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:07.843852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:07.843864 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:07.843924 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:07.844632 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.845380 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.846444 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.846782 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.846832 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:07.848372 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:07.848374 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:07.848375 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:07.848383 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:07.850417 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.850843 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.850845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.850846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.850912 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.851125 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.851125 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.851126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.851271 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:07.851362 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:07.851624 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 03:07:07.851720 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 03:07:07.852032 22761 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45785 } attempt: 3
I20260924 03:07:07.852164 22810 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f949abb18bff4316a7b9427409201be2 [term 1 FOLLOWER]: Refusing update from remote peer dffc328662d444589521930b74d45d4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:07:07.852680 22843 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f949abb18bff4316a7b9427409201be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:07:07.854094 22857 sys_catalog.cc:455] T 00000000000000000000000000000000 P f949abb18bff4316a7b9427409201be2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dffc328662d444589521930b74d45d4c. Latest consensus state: current_term: 1 leader_uuid: "dffc328662d444589521930b74d45d4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dffc328662d444589521930b74d45d4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36047 } } peers { permanent_uuid: "996dd202e939431683c274b79bddefde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40225 } } peers { permanent_uuid: "f949abb18bff4316a7b9427409201be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45785 } } }
I20260924 03:07:07.854193 22857 sys_catalog.cc:458] T 00000000000000000000000000000000 P f949abb18bff4316a7b9427409201be2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:07:07.856606 22869 sys_catalog.cc:455] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dffc328662d444589521930b74d45d4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dffc328662d444589521930b74d45d4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36047 } } peers { permanent_uuid: "996dd202e939431683c274b79bddefde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40225 } } peers { permanent_uuid: "f949abb18bff4316a7b9427409201be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45785 } } }
I20260924 03:07:07.856714 22869 sys_catalog.cc:458] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c [sys.catalog]: This master's current role is: LEADER
I20260924 03:07:07.857103 22857 sys_catalog.cc:455] T 00000000000000000000000000000000 P f949abb18bff4316a7b9427409201be2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dffc328662d444589521930b74d45d4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dffc328662d444589521930b74d45d4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36047 } } peers { permanent_uuid: "996dd202e939431683c274b79bddefde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40225 } } peers { permanent_uuid: "f949abb18bff4316a7b9427409201be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45785 } } }
I20260924 03:07:07.857182 22857 sys_catalog.cc:458] T 00000000000000000000000000000000 P f949abb18bff4316a7b9427409201be2 [sys.catalog]: This master's current role is: FOLLOWER
Sep 24 03:07:07 dist-test-slave-wgsb krb5kdc[22605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 03:07:07.857385 22877 mvcc.cc:205] Tried to move back new op lower bound from 7332737957235929088 to 7332737957191921664. Current Snapshot: MvccSnapshot[applied={T|T < 7332737957235929088}]
I20260924 03:07:07.857560 22881 mvcc.cc:205] Tried to move back new op lower bound from 7332737957235929088 to 7332737957191921664. Current Snapshot: MvccSnapshot[applied={T|T < 7332737957235929088}]
I20260924 03:07:07.861155 22763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.852304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45785 (local address 127.0.0.1:37932)
0924 03:07:07.852379 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:07.852390 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:07:07.852430 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:07:07.852701 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:07:07.852702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:07:07.852860 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:07:07.852893 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:07.852895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.853808 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.853809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:07.854622 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:07.854624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.855413 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.855414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:07.855464 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:07.856153 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:07:07.856158 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:07:07.857830 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:07:07.860017 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:07.860018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:07.860018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:07.860241 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:07.860451 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:07.860452 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:07.860452 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:07.860511 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:07.860824 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:07:07.860825 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:07:07.860872 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 03:07:07.860937 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 03:07:07.861045 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 03:07:07.861681 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.852473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37932 (local address 127.0.98.60:45785)
0924 03:07:07.852545 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:07.852548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:07.852556 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:07:07.852575 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:07.852576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:07.852608 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:07.852659 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:07.853064 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.853723 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.854857 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.855178 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.855350 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:07.857991 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:07.857992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:07.857993 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:07.858000 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:07.859923 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.860324 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.860326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.860326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.860381 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.860579 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.860580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.860580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.860724 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:07.860798 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:07.861506 (+ 708us) server_negotiation.cc:299] Negotiation successful
0924 03:07:07.861584 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 03:07:07.861708 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:07.840466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49236 (local address 127.0.98.61:40225)
0924 03:07:07.840520 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:07.840521 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:07.850837 (+ 10316us) server_negotiation.cc:407] Connection header received
0924 03:07:07.850985 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:07.850997 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:07.851012 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:07.851044 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:07.852662 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.853394 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.854931 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:07.855235 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:07.855276 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:07.857964 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:07.857965 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:07.857965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:07.857986 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:07.859930 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.860606 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.860607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.860608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.860666 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:07.861005 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:07.861006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:07.861006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:07.861118 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:07.861196 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:07.861562 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 03:07:07.861627 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 03:07:07.863778 22761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 996dd202e939431683c274b79bddefde: Bootstrap starting.
I20260924 03:07:07.864037 22875 catalog_manager.cc:1347] Generated new cluster ID: b99ad5ca87e8483595b5e63706d11882
I20260924 03:07:07.864097 22875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:07:07.864507 22761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 996dd202e939431683c274b79bddefde: Neither blocks nor log segments found. Creating new log.
I20260924 03:07:07.864817 22761 log.cc:824] T 00000000000000000000000000000000 P 996dd202e939431683c274b79bddefde: Log is configured to *not* fsync() on all Append() calls
I20260924 03:07:07.865613 22761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 996dd202e939431683c274b79bddefde: No bootstrap required, opened a new log
I20260924 03:07:07.868363 22761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 996dd202e939431683c274b79bddefde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dffc328662d444589521930b74d45d4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36047 } } peers { permanent_uuid: "996dd202e939431683c274b79bddefde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40225 } } peers { permanent_uuid: "f949abb18bff4316a7b9427409201be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45785 } }
I20260924 03:07:07.868536 22761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 996dd202e939431683c274b79bddefde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996dd202e939431683c274b79bddefde, State: Initialized, Role: FOLLOWER
I20260924 03:07:07.868626 22761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 996dd202e939431683c274b79bddefde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dffc328662d444589521930b74d45d4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36047 } } peers { permanent_uuid: "996dd202e939431683c274b79bddefde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40225 } } peers { permanent_uuid: "f949abb18bff4316a7b9427409201be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45785 } }
I20260924 03:07:07.868949 22886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 996dd202e939431683c274b79bddefde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dffc328662d444589521930b74d45d4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36047 } } peers { permanent_uuid: "996dd202e939431683c274b79bddefde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40225 } } peers { permanent_uuid: "f949abb18bff4316a7b9427409201be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45785 } } }
I20260924 03:07:07.869065 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dffc328662d444589521930b74d45d4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "996dd202e939431683c274b79bddefde"
I20260924 03:07:07.869215 22886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 996dd202e939431683c274b79bddefde [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:07:07.869057 22734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 996dd202e939431683c274b79bddefde [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:07:07.870260 22734 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 996dd202e939431683c274b79bddefde [term 1 FOLLOWER]: Refusing update from remote peer dffc328662d444589521930b74d45d4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:07:07.870682 22843 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "996dd202e939431683c274b79bddefde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:07:07.871984 22735 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 996dd202e939431683c274b79bddefde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dffc328662d444589521930b74d45d4c in term 1.
I20260924 03:07:07.868992 22761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 996dd202e939431683c274b79bddefde [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:07:07.875737 22901 catalog_manager.cc:1259] Loaded cluster ID: b99ad5ca87e8483595b5e63706d11882
I20260924 03:07:07.875854 22901 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 996dd202e939431683c274b79bddefde: loading cluster ID for follower catalog manager: success
W20260924 03:07:07.876009 22901 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 996dd202e939431683c274b79bddefde: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260924 03:07:07.876108 22901 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:07:07.876261 22886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 996dd202e939431683c274b79bddefde [sys.catalog]: SysCatalogTable state changed. Reason: New leader dffc328662d444589521930b74d45d4c. Latest consensus state: current_term: 1 leader_uuid: "dffc328662d444589521930b74d45d4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dffc328662d444589521930b74d45d4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36047 } } peers { permanent_uuid: "996dd202e939431683c274b79bddefde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40225 } } peers { permanent_uuid: "f949abb18bff4316a7b9427409201be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45785 } } }
I20260924 03:07:07.876333 22886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 996dd202e939431683c274b79bddefde [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:07:07.876377 22886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 996dd202e939431683c274b79bddefde [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dffc328662d444589521930b74d45d4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dffc328662d444589521930b74d45d4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36047 } } peers { permanent_uuid: "996dd202e939431683c274b79bddefde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40225 } } peers { permanent_uuid: "f949abb18bff4316a7b9427409201be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45785 } } }
I20260924 03:07:07.876427 22886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 996dd202e939431683c274b79bddefde [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:07:07.888828 22843 sys_catalog.cc:455] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dffc328662d444589521930b74d45d4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dffc328662d444589521930b74d45d4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36047 } } peers { permanent_uuid: "996dd202e939431683c274b79bddefde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40225 } } peers { permanent_uuid: "f949abb18bff4316a7b9427409201be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45785 } } }
I20260924 03:07:07.888967 22843 sys_catalog.cc:458] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c [sys.catalog]: This master's current role is: LEADER
I20260924 03:07:07.891626 22875 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:07:07.892817 22875 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:07:07.906629 22875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dffc328662d444589521930b74d45d4c: Generated new TSK 0
I20260924 03:07:07.907797 22875 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 03:07:07.938200 22765 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:07:07.936320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49252 (local address 127.0.98.61:40225)
0924 03:07:07.936370 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:07.936373 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:07.937362 (+ 989us) server_negotiation.cc:407] Connection header received
0924 03:07:07.937482 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:07.937483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:07.937513 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:07.937548 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:07.938117 (+ 569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49252: BlockingRecv error: recv got EOF from 127.0.0.1:49252 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260924 03:07:07.939049 22840 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:07:07.936732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37940 (local address 127.0.98.60:45785)
0924 03:07:07.936785 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:07.936787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:07.938476 (+ 1689us) server_negotiation.cc:407] Connection header received
0924 03:07:07.938585 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:07.938586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:07.938598 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:07.938631 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:07.938983 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37940: BlockingRecv error: recv got EOF from 127.0.0.1:37940 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260924 03:07:07.939877 22689 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:07:07.937103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40148 (local address 127.0.98.62:36047)
0924 03:07:07.937166 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:07.937168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:07.939288 (+ 2120us) server_negotiation.cc:407] Connection header received
0924 03:07:07.939395 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:07.939397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:07.939409 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:07.939440 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:07.939808 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40148: BlockingRecv error: recv got EOF from 127.0.0.1:40148 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260924 03:07:07.942093 22765 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:07:07.941338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49268 (local address 127.0.98.61:40225)
0924 03:07:07.941387 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:07.941389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:07.941528 (+ 139us) server_negotiation.cc:407] Connection header received
0924 03:07:07.941643 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:07.941644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:07.941664 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:07.941696 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:07.942040 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49268: BlockingRecv error: recv got EOF from 127.0.0.1:49268 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260924 03:07:07.945294 22765 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:07:07.944463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49284 (local address 127.0.98.61:40225)
0924 03:07:07.944502 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:07.944504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:07.944784 (+ 280us) server_negotiation.cc:407] Connection header received
0924 03:07:07.944887 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:07.944888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:07.944899 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:07.944930 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:07.945241 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49284: BlockingRecv error: recv got EOF from 127.0.0.1:49284 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 03:07:07.949384 22765 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:07:07.948489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49288 (local address 127.0.98.61:40225)
0924 03:07:07.948525 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:07.948527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:07.948803 (+ 276us) server_negotiation.cc:407] Connection header received
0924 03:07:07.948912 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:07.948913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:07.948924 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:07.948957 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:07.949331 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49288: BlockingRecv error: recv got EOF from 127.0.0.1:49288 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 03:07:07.956207 22765 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:07:07.955138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49300 (local address 127.0.98.61:40225)
0924 03:07:07.955185 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:07.955187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:07.955487 (+ 300us) server_negotiation.cc:407] Connection header received
0924 03:07:07.955695 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:07.955696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:07.955708 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:07.955741 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:07.956145 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49300: BlockingRecv error: recv got EOF from 127.0.0.1:49300 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 03:07:07.966645 22765 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:07:07.965548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49302 (local address 127.0.98.61:40225)
0924 03:07:07.965594 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:07.965597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:07.965924 (+ 327us) server_negotiation.cc:407] Connection header received
0924 03:07:07.966064 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:07.966066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:07.966078 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:07.966112 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:07.966546 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49302: BlockingRecv error: recv got EOF from 127.0.0.1:49302 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 03:07:07.986989 22765 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:07:07.985097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49310 (local address 127.0.98.61:40225)
0924 03:07:07.985146 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:07.985148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:07.986024 (+ 876us) server_negotiation.cc:407] Connection header received
0924 03:07:07.986154 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:07.986155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:07.986169 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:07.986204 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:07.986917 (+ 713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49310: BlockingRecv error: recv got EOF from 127.0.0.1:49310 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 03:07:08.021747 22765 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:07:08.020689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49316 (local address 127.0.98.61:40225)
0924 03:07:08.020738 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:08.020741 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:08.021082 (+ 341us) server_negotiation.cc:407] Connection header received
0924 03:07:08.021201 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:08.021203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:08.021216 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:08.021250 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:08.021679 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49316: BlockingRecv error: recv got EOF from 127.0.0.1:49316 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 03:07:08.087981 22765 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:07:08.087112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49324 (local address 127.0.98.61:40225)
0924 03:07:08.087254 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:08.087257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:08.087407 (+ 150us) server_negotiation.cc:407] Connection header received
0924 03:07:08.087491 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:08.087492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:08.087504 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:08.087556 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:08.087910 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49324: BlockingRecv error: recv got EOF from 127.0.0.1:49324 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260924 03:07:08.218456 22765 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:07:08.217689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49334 (local address 127.0.98.61:40225)
0924 03:07:08.217783 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:08.217785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:08.217794 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:07:08.217821 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:08.217822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:08.217833 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:08.217886 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:08.218377 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49334: BlockingRecv error: recv got EOF from 127.0.0.1:49334 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 03:07:08.477439 22765 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:07:08.475965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49346 (local address 127.0.98.61:40225)
0924 03:07:08.476051 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:08.476053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:08.476063 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:07:08.476090 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:08.476090 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:08.476101 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:08.476559 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:08.477358 (+ 799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49346: BlockingRecv error: recv got EOF from 127.0.0.1:49346 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 03:07:08.832307 22868 catalog_manager.cc:1259] Loaded cluster ID: b99ad5ca87e8483595b5e63706d11882
I20260924 03:07:08.832391 22868 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f949abb18bff4316a7b9427409201be2: loading cluster ID for follower catalog manager: success
I20260924 03:07:08.833199 22868 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f949abb18bff4316a7b9427409201be2: acquiring CA information for follower catalog manager: success
I20260924 03:07:08.833477 22868 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f949abb18bff4316a7b9427409201be2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:07:08.877218 22901 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 996dd202e939431683c274b79bddefde: acquiring CA information for follower catalog manager: success
I20260924 03:07:08.877652 22901 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 996dd202e939431683c274b79bddefde: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:07:08.993731 22967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:08.990793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49362 (local address 127.0.98.61:40225)
0924 03:07:08.990894 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:08.990896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:08.990905 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:07:08.990994 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:08.990995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:08.991044 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:08.991120 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:07:08.991358 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:08.991950 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:08.992798 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:08.992991 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:08.993048 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:08.993239 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:07:08.993414 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:07:08.993487 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:08.993607 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 03:07:08.993626 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":54,"threads_started":1}
I20260924 03:07:08.998373 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:08.996014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37948 (local address 127.0.98.60:45785)
0924 03:07:08.996137 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:08.996139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:08.996148 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:07:08.996183 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:08.996183 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:08.996217 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:08.996302 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:07:08.996508 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:08.996985 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:08.997727 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:08.997907 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:08.997956 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:08.998081 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:07:08.998176 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:07:08.998233 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:08.998268 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 03:07:08.998283 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":74,"threads_started":1}
I20260924 03:07:09.003302 22967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:09.000220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49370 (local address 127.0.98.61:40225)
0924 03:07:09.000271 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:09.000273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:09.000448 (+ 175us) server_negotiation.cc:407] Connection header received
0924 03:07:09.000562 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:09.000563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:09.000576 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:09.000604 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:07:09.000756 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:09.001515 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:09.002421 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:09.002710 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:09.002748 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:09.002985 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:07:09.003021 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:07:09.003070 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:09.003226 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 03:07:09.003243 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 03:07:09.003319 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:09.000210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40160 (local address 127.0.98.62:36047)
0924 03:07:09.000321 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:09.000323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:09.000333 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:07:09.000453 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:09.000455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:09.000512 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:09.000593 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:07:09.000697 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:09.001262 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:09.002259 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:09.002572 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:09.002630 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:09.002851 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:07:09.002970 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:07:09.003032 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:09.003186 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 03:07:09.003214 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":52,"threads_started":1}
I20260924 03:07:09.003679 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:09.000463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37958 (local address 127.0.98.60:45785)
0924 03:07:09.000498 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:09.000500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:09.000534 (+ 34us) server_negotiation.cc:407] Connection header received
0924 03:07:09.000630 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:09.000631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:09.000641 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:09.000667 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:07:09.000838 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:09.001545 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:09.002598 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:09.002840 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:09.002904 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:09.003076 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:07:09.003105 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:07:09.003159 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:09.003576 (+ 417us) server_negotiation.cc:299] Negotiation successful
0924 03:07:09.003598 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:07:09.005100 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 22618
I20260924 03:07:09.011075 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 22693
I20260924 03:07:09.016461 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 22768
2026-09-24T03:07:09Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2915 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:07:09 dist-test-slave-wgsb krb5kdc[22984](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:07:09 dist-test-slave-wgsb krb5kdc[22984](info): set up 2 sockets
Sep 24 03:07:09 dist-test-slave-wgsb krb5kdc[22984](info): commencing operation
krb5kdc: starting...
W20260924 03:07:10.064109 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/krb5kdc/test-user.keytab.
Sep 24 03:07:10 dist-test-slave-wgsb krb5kdc[22984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:07:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:07:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:07:10.195091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35663
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35663,127.0.98.61:46079,127.0.98.60:43349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:07:10.289661 22997 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:07:10.290021 22997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:07:10.290081 22997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:07:10.290145 22997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:07:10.291903 22997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:07:10.291970 22997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:07:10.291991 22997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:07:10.292009 22997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:07:10.292027 22997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:07:10.293845 22997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35663,127.0.98.61:46079,127.0.98.60:43349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:07:10.294112 22997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:07:10.294379 22997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:07:10.300467 23003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:07:10.300696 22997 server_base.cc:1034] running on GCE node
W20260924 03:07:10.300467 23002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:07:10.304582 23005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:07:10.304952 22997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:07:10.305284 22997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:07:10.306481 22997 hybrid_clock.cc:648] HybridClock initialized: now 1790219230306458 us; error 37 us; skew 500 ppm
Sep 24 03:07:10 dist-test-slave-wgsb krb5kdc[22984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:07:10.309399 22997 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:07:10.309899 22997 webserver.cc:460] Webserver started at http://127.0.98.62:39063/ using document root <none> and password file <none>
I20260924 03:07:10.310148 22997 fs_manager.cc:359] Metadata directory not provided
I20260924 03:07:10.310225 22997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:07:10.310391 22997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:07:10.311569 22997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-0/data/instance:
uuid: "e459bd18b0084b9dbe2b19d6344eb22d"
format_stamp: "Formatted at 2026-09-24 03:07:10 on dist-test-slave-wgsb"
server_key: "96e97f36a0c95eab3dd209e0174fd303"
server_key_iv: "8f8345e3122bbe5036cac5b7308b11cf"
server_key_version: "encryptionkey@0"
I20260924 03:07:10.312045 22997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-0/wal/instance:
uuid: "e459bd18b0084b9dbe2b19d6344eb22d"
format_stamp: "Formatted at 2026-09-24 03:07:10 on dist-test-slave-wgsb"
server_key: "96e97f36a0c95eab3dd209e0174fd303"
server_key_iv: "8f8345e3122bbe5036cac5b7308b11cf"
server_key_version: "encryptionkey@0"
I20260924 03:07:10.313514 22997 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:07:10.314277 23012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:07:10.314553 22997 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:07:10.314661 22997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-0/wal
uuid: "e459bd18b0084b9dbe2b19d6344eb22d"
format_stamp: "Formatted at 2026-09-24 03:07:10 on dist-test-slave-wgsb"
server_key: "96e97f36a0c95eab3dd209e0174fd303"
server_key_iv: "8f8345e3122bbe5036cac5b7308b11cf"
server_key_version: "encryptionkey@0"
I20260924 03:07:10.314756 22997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:07:10.329013 22997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:07:10.331355 22997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:07:10.331585 22997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:07:10.335880 22997 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35663
I20260924 03:07:10.335922 23064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35663 every 8 connection(s)
I20260924 03:07:10.336293 22997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:07:10.337414 23065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:07:10.339099 23065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35663 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:07:10.340320 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 22997
I20260924 03:07:10.340411 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-0/wal/instance
I20260924 03:07:10.340631 392 external_mini_cluster.cc:1428] Setting key bcc3551c8ae3748117f823ca3d65f929
Sep 24 03:07:10 dist-test-slave-wgsb krb5kdc[22984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260924 03:07:10.351845 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.339939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50796 (local address 127.0.98.62:35663)
0924 03:07:10.340162 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:10.340172 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:10.340929 (+ 757us) server_negotiation.cc:407] Connection header received
0924 03:07:10.341551 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:10.341553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:10.341574 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:10.341682 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:10.342304 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.343422 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.344553 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.345032 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.345154 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:10.348035 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:10.348043 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:10.348043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:10.348050 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:10.349981 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.350430 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.350433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.350433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.350524 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.350791 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.350792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.350793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.351089 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:10.351196 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:10.351423 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 03:07:10.351572 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":135,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260924 03:07:10.351847 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.339912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35663 (local address 127.0.0.1:50796)
0924 03:07:10.340748 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:10.340814 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 03:07:10.341387 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:07:10.341688 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:07:10.341691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:07:10.342038 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:07:10.342194 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:10.342199 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.343560 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.343562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:10.344459 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:10.344462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.345151 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.345153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:10.345252 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:10.345952 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:07:10.345978 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:07:10.347916 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:07:10.350094 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:10.350100 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:10.350102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:10.350317 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:10.350599 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:10.350600 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:10.350600 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:10.350702 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:10.351192 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:07:10.351197 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:07:10.351267 (+ 70us) client_negotiation.cc:769] Sending connection context
0924 03:07:10.351383 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 03:07:10.351738 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":104,"thread_start_us":111,"threads_started":1}
I20260924 03:07:10.353118 23065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46079 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:07:10.353538 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.353387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46079 (local address 127.0.0.1:49930)
0924 03:07:10.353442 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:10.353499 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 03:07:10.353700 23013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46079: connect: Connection refused (error 111)
I20260924 03:07:10.354174 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.354049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46079 (local address 127.0.0.1:49940)
0924 03:07:10.354107 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:10.354144 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 03:07:10.354365 23065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46079: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46079: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 03:07:10.370816 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46079
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35663,127.0.98.61:46079,127.0.98.60:43349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:07:10.403645 23065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46079 } attempt: 1
I20260924 03:07:10.404150 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.404014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46079 (local address 127.0.0.1:49942)
0924 03:07:10.404080 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:10.404121 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 03:07:10.404515 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.404406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46079 (local address 127.0.0.1:49954)
0924 03:07:10.404468 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:10.404497 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 03:07:10.404603 23065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46079: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46079: connect: Connection refused (error 111)
W20260924 03:07:10.454260 23072 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:07:10.454521 23072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:07:10.454552 23072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:07:10.454599 23072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:07:10.455886 23072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:07:10.455948 23072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:07:10.455972 23072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:07:10.456004 23072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:07:10.456032 23072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:07:10.457520 23072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35663,127.0.98.61:46079,127.0.98.60:43349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:07:10.457794 23072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:07:10.458025 23072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:07:10.462841 23080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:07:10.462972 23077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:07:10.462972 23078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:07:10.462994 23072 server_base.cc:1034] running on GCE node
I20260924 03:07:10.463384 23072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:07:10.463739 23072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:07:10.464905 23072 hybrid_clock.cc:648] HybridClock initialized: now 1790219230464883 us; error 38 us; skew 500 ppm
Sep 24 03:07:10 dist-test-slave-wgsb krb5kdc[22984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:07:10.467679 23072 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 03:07:10.468152 23072 webserver.cc:460] Webserver started at http://127.0.98.61:39221/ using document root <none> and password file <none>
I20260924 03:07:10.468365 23072 fs_manager.cc:359] Metadata directory not provided
I20260924 03:07:10.468406 23072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:07:10.468495 23072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:07:10.469519 23072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-1/data/instance:
uuid: "9a6aa4eccbf14acd842659653297f685"
format_stamp: "Formatted at 2026-09-24 03:07:10 on dist-test-slave-wgsb"
server_key: "3b7c00e05e9949dabba54a4273ec73b4"
server_key_iv: "1e7d00ca2d79f24a866bca6c444d2e5a"
server_key_version: "encryptionkey@0"
I20260924 03:07:10.469815 23072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-1/wal/instance:
uuid: "9a6aa4eccbf14acd842659653297f685"
format_stamp: "Formatted at 2026-09-24 03:07:10 on dist-test-slave-wgsb"
server_key: "3b7c00e05e9949dabba54a4273ec73b4"
server_key_iv: "1e7d00ca2d79f24a866bca6c444d2e5a"
server_key_version: "encryptionkey@0"
I20260924 03:07:10.471184 23072 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:07:10.472173 23087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:07:10.472383 23072 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 03:07:10.472462 23072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-1/wal
uuid: "9a6aa4eccbf14acd842659653297f685"
format_stamp: "Formatted at 2026-09-24 03:07:10 on dist-test-slave-wgsb"
server_key: "3b7c00e05e9949dabba54a4273ec73b4"
server_key_iv: "1e7d00ca2d79f24a866bca6c444d2e5a"
server_key_version: "encryptionkey@0"
I20260924 03:07:10.472513 23072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:07:10.472718 23065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46079 } attempt: 2
I20260924 03:07:10.473112 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.472990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46079 (local address 127.0.0.1:49966)
0924 03:07:10.473048 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:10.473084 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 03:07:10.473570 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.473507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46079 (local address 127.0.0.1:49976)
0924 03:07:10.473532 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:10.473555 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260924 03:07:10.473671 23065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46079: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46079: connect: Connection refused (error 111)
I20260924 03:07:10.490780 23072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:07:10.492981 23072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:07:10.493141 23072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:07:10.500030 23072 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46079
I20260924 03:07:10.500291 23139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46079 every 8 connection(s)
I20260924 03:07:10.500902 23072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-1/data/info.pb
I20260924 03:07:10.502272 23140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:07:10.504026 23140 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35663 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:07:10.506218 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 23072
I20260924 03:07:10.506386 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-1/wal/instance
I20260924 03:07:10.506585 392 external_mini_cluster.cc:1428] Setting key 11562aca74b363f0918f606859c6599e
Sep 24 03:07:10 dist-test-slave-wgsb krb5kdc[22984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:07:10.514194 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.504825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50810 (local address 127.0.98.62:35663)
0924 03:07:10.504938 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:10.504941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:10.505109 (+ 168us) server_negotiation.cc:407] Connection header received
0924 03:07:10.505664 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:10.505665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:10.505680 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:10.505749 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:10.506461 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.507009 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.508088 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.508313 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.508401 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:10.510810 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:10.510811 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:10.510812 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:10.510821 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:10.512616 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.513042 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.513044 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.513044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.513111 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.513427 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.513429 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.513429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.513582 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:10.513662 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:10.513969 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 03:07:10.514062 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 03:07:10.514235 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.504719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35663 (local address 127.0.0.1:50810)
0924 03:07:10.504986 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:10.505025 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 03:07:10.505547 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:07:10.505855 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:07:10.505858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:07:10.506131 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:07:10.506357 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:10.506363 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.507115 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.507116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:10.507999 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:10.508001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.508427 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.508429 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:10.508557 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:10.509064 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:07:10.509072 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:07:10.510704 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:07:10.512708 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:10.512714 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:10.512716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:10.512961 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:10.513179 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:10.513180 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:10.513180 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:10.513355 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:10.513678 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:07:10.513684 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:07:10.513820 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 03:07:10.513930 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 03:07:10.514064 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260924 03:07:10.515089 23140 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46079 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 03:07:10 dist-test-slave-wgsb krb5kdc[22984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:07:10.524760 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.515388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46079 (local address 127.0.0.1:49978)
0924 03:07:10.515451 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:10.515463 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:07:10.515509 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:07:10.515944 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:07:10.515945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:07:10.516123 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:07:10.516180 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:10.516181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.517116 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.517117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:10.517713 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:10.517715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.518256 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.518257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:10.518341 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:10.518926 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:07:10.518932 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:07:10.520750 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:07:10.523065 (+ 2315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:10.523066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:10.523066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:10.523296 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:10.523568 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:10.523569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:10.523570 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:10.523687 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:10.524138 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:07:10.524139 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:07:10.524227 (+ 88us) client_negotiation.cc:769] Sending connection context
0924 03:07:10.524343 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 03:07:10.524525 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 03:07:10.524838 23144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.515429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49978 (local address 127.0.98.61:46079)
0924 03:07:10.515624 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:10.515628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:10.515647 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:07:10.515671 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:10.515677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:10.515689 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:10.515752 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:10.516287 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.517056 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.517775 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.518130 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.518240 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:10.520862 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:10.520870 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:10.520870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:10.520877 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:10.522962 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.523384 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.523388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.523389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.523484 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.523825 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.523826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.523826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.524055 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:10.524154 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:10.524396 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 03:07:10.524650 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":119,"server-negotiator.queue_time_us":146,"thread_start_us":119,"threads_started":1}
I20260924 03:07:10.525820 23140 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43349 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:07:10.526275 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.526088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43349 (local address 127.0.0.1:45468)
0924 03:07:10.526161 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:10.526244 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 03:07:10.526571 23090 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43349: connect: Connection refused (error 111)
I20260924 03:07:10.527701 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.527591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43349 (local address 127.0.0.1:45480)
0924 03:07:10.527649 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:10.527677 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 03:07:10.527824 23140 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43349: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43349: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 03:07:10.539935 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskptPCbv/build/release/bin/kudu
/tmp/dist-test-taskptPCbv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43349
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35663,127.0.98.61:46079,127.0.98.60:43349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:07:10.564812 23065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46079 } attempt: 3
Sep 24 03:07:10 dist-test-slave-wgsb krb5kdc[22984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:07:10.572232 23144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.565303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49988 (local address 127.0.98.61:46079)
0924 03:07:10.565376 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:10.565379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:10.565392 (+ 13us) server_negotiation.cc:407] Connection header received
0924 03:07:10.565469 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:10.565470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:10.565483 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:10.565546 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:10.565859 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.566630 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.567334 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.567676 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.567749 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:10.569713 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:10.569715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:10.569715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:10.569723 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:10.571122 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.571502 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.571503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.571503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.571552 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.571778 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.571779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.571779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.571883 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:10.571937 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:10.572102 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 03:07:10.572164 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 03:07:10.572270 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.565225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46079 (local address 127.0.0.1:49988)
0924 03:07:10.565316 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:10.565330 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:07:10.565396 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:07:10.565559 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:07:10.565559 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:07:10.565703 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:07:10.565785 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:10.565787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.566717 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.566717 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:10.567252 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:10.567253 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.567748 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.567748 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:10.567809 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:10.568292 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:07:10.568296 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:07:10.569621 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:07:10.571208 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:10.571209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:10.571209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:10.571425 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:10.571645 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:10.571646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:10.571646 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:10.571714 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:10.571949 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:07:10.571954 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:07:10.572002 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 03:07:10.572076 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 03:07:10.572172 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 03:07:10.572808 23065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43349 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:07:10.573177 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.573078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43349 (local address 127.0.0.1:45484)
0924 03:07:10.573128 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:10.573157 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 03:07:10.573529 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.573392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43349 (local address 127.0.0.1:45486)
0924 03:07:10.573488 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:10.573510 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260924 03:07:10.573737 23065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43349: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43349: connect: Connection refused (error 111)
I20260924 03:07:10.577026 23140 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43349 } attempt: 1
I20260924 03:07:10.577369 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.577248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43349 (local address 127.0.0.1:45498)
0924 03:07:10.577302 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:10.577338 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 03:07:10.577755 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.577662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43349 (local address 127.0.0.1:45508)
0924 03:07:10.577710 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:10.577736 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 03:07:10.577884 23140 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43349: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43349: connect: Connection refused (error 111)
I20260924 03:07:10.604907 23065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43349 } attempt: 1
I20260924 03:07:10.605460 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.605280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43349 (local address 127.0.0.1:45522)
0924 03:07:10.605349 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:10.605392 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 03:07:10.605796 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.605694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43349 (local address 127.0.0.1:45538)
0924 03:07:10.605741 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:10.605767 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 03:07:10.605885 23065 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43349: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43349: connect: Connection refused (error 111)
W20260924 03:07:10.625365 23147 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:07:10.625576 23147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:07:10.625594 23147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:07:10.625614 23147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:07:10.626791 23147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:07:10.626835 23147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:07:10.626852 23147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:07:10.626863 23147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:07:10.626875 23147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:07:10.628374 23147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35663,127.0.98.61:46079,127.0.98.60:43349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:07:10.628621 23147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:07:10.628898 23147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:07:10.633868 23155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:07:10.634016 23152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:07:10.633934 23153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:07:10.634457 23147 server_base.cc:1034] running on GCE node
I20260924 03:07:10.634720 23147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:07:10.634979 23147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:07:10.636142 23147 hybrid_clock.cc:648] HybridClock initialized: now 1790219230636128 us; error 28 us; skew 500 ppm
Sep 24 03:07:10 dist-test-slave-wgsb krb5kdc[22984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:07:10.638729 23147 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 03:07:10.639245 23147 webserver.cc:460] Webserver started at http://127.0.98.60:38141/ using document root <none> and password file <none>
I20260924 03:07:10.639433 23147 fs_manager.cc:359] Metadata directory not provided
I20260924 03:07:10.639474 23147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:07:10.639564 23147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:07:10.640668 23147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-2/data/instance:
uuid: "b13b5707bade43958f578a3e7d1bc241"
format_stamp: "Formatted at 2026-09-24 03:07:10 on dist-test-slave-wgsb"
server_key: "994c23d5e910942fc1cbe5fdb32c96a6"
server_key_iv: "c153547583075df836b31c106bc7d5fe"
server_key_version: "encryptionkey@0"
I20260924 03:07:10.640981 23147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-2/wal/instance:
uuid: "b13b5707bade43958f578a3e7d1bc241"
format_stamp: "Formatted at 2026-09-24 03:07:10 on dist-test-slave-wgsb"
server_key: "994c23d5e910942fc1cbe5fdb32c96a6"
server_key_iv: "c153547583075df836b31c106bc7d5fe"
server_key_version: "encryptionkey@0"
I20260924 03:07:10.642269 23147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:07:10.642994 23162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:07:10.643177 23147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:07:10.643277 23147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-2/wal
uuid: "b13b5707bade43958f578a3e7d1bc241"
format_stamp: "Formatted at 2026-09-24 03:07:10 on dist-test-slave-wgsb"
server_key: "994c23d5e910942fc1cbe5fdb32c96a6"
server_key_iv: "c153547583075df836b31c106bc7d5fe"
server_key_version: "encryptionkey@0"
I20260924 03:07:10.643360 23147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:07:10.646051 23140 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43349 } attempt: 2
I20260924 03:07:10.646495 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.646373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43349 (local address 127.0.0.1:45548)
0924 03:07:10.646431 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:10.646469 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 03:07:10.646845 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.646733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43349 (local address 127.0.0.1:45562)
0924 03:07:10.646777 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:10.646828 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 03:07:10.647006 23140 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43349: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43349: connect: Connection refused (error 111)
I20260924 03:07:10.672544 23147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:07:10.674775 23147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:07:10.674947 23147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:07:10.679438 23147 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43349
I20260924 03:07:10.679474 23214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43349 every 8 connection(s)
I20260924 03:07:10.679888 23147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-2/data/info.pb
I20260924 03:07:10.680954 23215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:07:10.681036 23065 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43349 } attempt: 2
I20260924 03:07:10.683079 23215 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35663 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:07:10.685389 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskptPCbv/build/release/bin/kudu as pid 23147
I20260924 03:07:10.685493 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskptPCbv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218948286999-392-0/minicluster-data/master-2/wal/instance
I20260924 03:07:10.685684 392 external_mini_cluster.cc:1428] Setting key b36609ffc33abe05ebe1cfd79906bc8c
Sep 24 03:07:10 dist-test-slave-wgsb krb5kdc[22984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 03:07:10 dist-test-slave-wgsb krb5kdc[22984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:07:10.691661 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.681439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43349 (local address 127.0.0.1:45570)
0924 03:07:10.681507 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:10.681519 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:07:10.681567 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:07:10.682628 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:07:10.682629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:07:10.682824 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:07:10.682867 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:10.682869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.684285 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.684287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:10.685055 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:10.685056 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.685934 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.685936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:10.686007 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:10.686439 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:07:10.686444 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:07:10.688036 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:07:10.690175 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:10.690176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:10.690177 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:10.690443 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:10.690707 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:10.690708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:10.690709 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:10.690778 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:10.691232 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:07:10.691233 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:07:10.691279 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 03:07:10.691426 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 03:07:10.691526 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
Sep 24 03:07:10 dist-test-slave-wgsb krb5kdc[22984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 03:07:10.691923 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.681578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45570 (local address 127.0.98.60:43349)
0924 03:07:10.681773 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:10.681782 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:10.681809 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:07:10.682438 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:10.682444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:10.682490 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:10.682640 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:10.682970 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.684190 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.685129 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.685801 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.685979 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:10.688119 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:10.688126 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:10.688127 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:10.688133 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:10.690096 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.690543 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.690546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.690546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.690638 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.690842 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.690843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.690843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.691140 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:10.691244 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:10.691509 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 03:07:10.691674 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":86,"threads_started":1}
I20260924 03:07:10.692341 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.683786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50814 (local address 127.0.98.62:35663)
0924 03:07:10.683849 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:10.683851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:10.683980 (+ 129us) server_negotiation.cc:407] Connection header received
0924 03:07:10.684045 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:10.684046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:10.684055 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:10.684113 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:10.684547 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.685183 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.686253 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.686567 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.686669 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:10.689071 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:10.689072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:10.689072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:10.689080 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:10.690818 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.691201 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.691202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.691203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.691260 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.691488 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.691489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.691490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.691641 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:10.691687 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:10.692185 (+ 498us) server_negotiation.cc:299] Negotiation successful
0924 03:07:10.692260 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 03:07:10.692672 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.683719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35663 (local address 127.0.0.1:50814)
0924 03:07:10.683881 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:10.683918 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:07:10.683996 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:07:10.684143 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:07:10.684145 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:07:10.684407 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:07:10.684470 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:10.684472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.685284 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.685285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:10.686166 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:10.686168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.686701 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.686703 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:10.686782 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:10.687235 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:07:10.687245 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:07:10.689002 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:07:10.690901 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:10.690905 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:10.690907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:10.691121 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:10.691349 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:10.691350 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:10.691351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:10.691410 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:10.691986 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:07:10.691991 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:07:10.692046 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 03:07:10.692464 (+ 418us) client_negotiation.cc:240] Negotiation successful
0924 03:07:10.692579 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":108,"threads_started":1}
I20260924 03:07:10.693240 23215 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46079 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:07:10.693822 23065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d: Bootstrap starting.
I20260924 03:07:10.694850 23065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d: Neither blocks nor log segments found. Creating new log.
I20260924 03:07:10.695169 23065 log.cc:824] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d: Log is configured to *not* fsync() on all Append() calls
I20260924 03:07:10.696095 23065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d: No bootstrap required, opened a new log
I20260924 03:07:10.696307 23220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.686110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45586 (local address 127.0.98.60:43349)
0924 03:07:10.686540 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:10.686542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:10.686550 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:07:10.686569 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:10.686569 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:10.686578 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:10.687352 (+ 774us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:10.687409 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.688120 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.689241 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.689559 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.689631 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:10.692494 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:10.692496 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:10.692497 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:10.692504 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:10.694964 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.695317 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.695318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.695318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.695376 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.695585 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.695586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.695586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.695754 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:10.695916 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:10.696126 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 03:07:10.696209 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":361,"threads_started":1}
Sep 24 03:07:10 dist-test-slave-wgsb krb5kdc[22984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:07:10.698086 23065 raft_consensus.cc:348] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35663 } } peers { permanent_uuid: "9a6aa4eccbf14acd842659653297f685" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46079 } } peers { permanent_uuid: "b13b5707bade43958f578a3e7d1bc241" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43349 } }
I20260924 03:07:10.698215 23065 raft_consensus.cc:729] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e459bd18b0084b9dbe2b19d6344eb22d, State: Initialized, Role: FOLLOWER
I20260924 03:07:10.698283 23065 consensus_queue.cc:260] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35663 } } peers { permanent_uuid: "9a6aa4eccbf14acd842659653297f685" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46079 } } peers { permanent_uuid: "b13b5707bade43958f578a3e7d1bc241" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43349 } }
I20260924 03:07:10.698612 23065 sys_catalog.cc:564] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:07:10.698575 23222 sys_catalog.cc:455] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35663 } } peers { permanent_uuid: "9a6aa4eccbf14acd842659653297f685" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46079 } } peers { permanent_uuid: "b13b5707bade43958f578a3e7d1bc241" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43349 } } }
I20260924 03:07:10.698668 23222 sys_catalog.cc:458] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d [sys.catalog]: This master's current role is: FOLLOWER
W20260924 03:07:10.700577 23233 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:07:10.700634 23233 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:07:10.701150 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.693535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46079 (local address 127.0.0.1:49990)
0924 03:07:10.693576 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:10.693586 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:07:10.693623 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:07:10.693815 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:07:10.693816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:07:10.693999 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:07:10.694050 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:10.694053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.695122 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.695122 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:10.695934 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:10.695937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.696508 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.696508 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:10.696582 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:10.697008 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:07:10.697014 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:07:10.698265 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:07:10.700057 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:10.700058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:10.700059 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:10.700261 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:10.700510 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:10.700511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:10.700511 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:10.700571 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:10.700866 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:07:10.700866 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:07:10.700910 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 03:07:10.700972 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 03:07:10.701067 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 03:07:10.701447 23144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.693631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49990 (local address 127.0.98.61:46079)
0924 03:07:10.693685 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:10.693687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:10.693695 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:07:10.693721 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:10.693721 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:10.693731 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:10.693798 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:10.694197 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.695012 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.696031 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.696377 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.696427 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:10.698367 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:10.698369 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:10.698369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:10.698377 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:10.699967 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.700359 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.700361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.700361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.700425 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.700639 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.700640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.700640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.700779 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:10.700839 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:10.701254 (+ 415us) server_negotiation.cc:299] Negotiation successful
0924 03:07:10.701353 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 03:07:10.701818 23215 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43349 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:07:10 dist-test-slave-wgsb krb5kdc[22984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:07:10.701411 22997 master_runner.cc:186] Error getting master registration for 127.0.98.61:46079: OK, instance_id { permanent_uuid: "9a6aa4eccbf14acd842659653297f685" instance_seqno: 1790219230496576 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:07:10.704756 22997 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:07:10.706332 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.697020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50826 (local address 127.0.98.62:35663)
0924 03:07:10.697059 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:10.697060 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:10.697068 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:07:10.697089 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:10.697090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:10.697099 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:10.697161 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:10.697744 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.698425 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.700069 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.700323 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.700368 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:10.703452 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:10.703454 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:10.703454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:10.703461 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:10.705119 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.705520 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.705521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.705521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.705583 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.705798 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.705799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.705800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.705911 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:10.706063 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:10.706148 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 03:07:10.706222 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 24 03:07:10 dist-test-slave-wgsb krb5kdc[22984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 03:07:10.709992 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.702084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43349 (local address 127.0.0.1:45600)
0924 03:07:10.702123 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:10.702132 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:07:10.702166 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:07:10.702440 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:07:10.702441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:07:10.702576 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:07:10.702607 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:10.702609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.703465 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.703466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:10.704775 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:10.704778 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.705259 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.705260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:10.705306 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:10.705744 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:07:10.705749 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:07:10.707100 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:07:10.708928 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:10.708929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:10.708929 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:10.709121 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:10.709324 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:10.709325 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:10.709325 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:10.709393 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:10.709727 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:07:10.709728 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:07:10.709762 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 03:07:10.709807 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 03:07:10.709902 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 03:07:10.710340 23220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.702236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45600 (local address 127.0.98.60:43349)
0924 03:07:10.702326 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:10.702328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:10.702336 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:07:10.702355 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:10.702356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:10.702365 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:10.702669 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:10.702690 (+ 21us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.703375 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.704864 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.705182 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.705591 (+ 409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:10.707184 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:10.707186 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:10.707187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:10.707195 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:10.708825 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.709193 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.709193 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.709193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.709237 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.709472 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.709473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.709473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.709647 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:10.710158 (+ 511us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:10.710197 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 03:07:10.710264 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 03:07:10.711817 23215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b13b5707bade43958f578a3e7d1bc241: Bootstrap starting.
Sep 24 03:07:10 dist-test-slave-wgsb krb5kdc[22984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:07:10.712646 23215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b13b5707bade43958f578a3e7d1bc241: Neither blocks nor log segments found. Creating new log.
I20260924 03:07:10.712955 23215 log.cc:824] T 00000000000000000000000000000000 P b13b5707bade43958f578a3e7d1bc241: Log is configured to *not* fsync() on all Append() calls
I20260924 03:07:10.713840 23215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b13b5707bade43958f578a3e7d1bc241: No bootstrap required, opened a new log
I20260924 03:07:10.714951 23144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.707414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50004 (local address 127.0.98.61:46079)
0924 03:07:10.707455 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:10.707457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:10.707694 (+ 237us) server_negotiation.cc:407] Connection header received
0924 03:07:10.707752 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:10.707753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:10.707764 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:10.707819 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:10.708392 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.708852 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.709607 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.709828 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.709902 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:10.712344 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:10.712346 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:10.712346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:10.712354 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:10.713832 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.714187 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.714188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.714189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.714241 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.714448 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.714449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.714449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.714563 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:10.714712 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:10.714781 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 03:07:10.714850 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 03:07:10.715839 23215 raft_consensus.cc:348] T 00000000000000000000000000000000 P b13b5707bade43958f578a3e7d1bc241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35663 } } peers { permanent_uuid: "9a6aa4eccbf14acd842659653297f685" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46079 } } peers { permanent_uuid: "b13b5707bade43958f578a3e7d1bc241" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43349 } }
I20260924 03:07:10.716038 23215 raft_consensus.cc:729] T 00000000000000000000000000000000 P b13b5707bade43958f578a3e7d1bc241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b13b5707bade43958f578a3e7d1bc241, State: Initialized, Role: FOLLOWER
I20260924 03:07:10.716115 23215 consensus_queue.cc:260] T 00000000000000000000000000000000 P b13b5707bade43958f578a3e7d1bc241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35663 } } peers { permanent_uuid: "9a6aa4eccbf14acd842659653297f685" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46079 } } peers { permanent_uuid: "b13b5707bade43958f578a3e7d1bc241" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43349 } }
I20260924 03:07:10.716240 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:07:10.716404 23215 sys_catalog.cc:564] T 00000000000000000000000000000000 P b13b5707bade43958f578a3e7d1bc241 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:07:10.716387 23236 sys_catalog.cc:455] T 00000000000000000000000000000000 P b13b5707bade43958f578a3e7d1bc241 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35663 } } peers { permanent_uuid: "9a6aa4eccbf14acd842659653297f685" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46079 } } peers { permanent_uuid: "b13b5707bade43958f578a3e7d1bc241" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43349 } } }
I20260924 03:07:10.716563 23236 sys_catalog.cc:458] T 00000000000000000000000000000000 P b13b5707bade43958f578a3e7d1bc241 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:07:10.717150 23039 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e459bd18b0084b9dbe2b19d6344eb22d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50826
I20260924 03:07:10.717298 23039 raft_consensus.cc:482] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 03:07:10.717360 23039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:07:10.718343 23039 raft_consensus.cc:504] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35663 } } peers { permanent_uuid: "9a6aa4eccbf14acd842659653297f685" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46079 } } peers { permanent_uuid: "b13b5707bade43958f578a3e7d1bc241" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43349 } }
W20260924 03:07:10.718513 23247 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b13b5707bade43958f578a3e7d1bc241: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 03:07:10.718613 23039 leader_election.cc:288] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d [CANDIDATE]: Term 1 election: Requested vote from peers 9a6aa4eccbf14acd842659653297f685 (127.0.98.61:46079), b13b5707bade43958f578a3e7d1bc241 (127.0.98.60:43349)
W20260924 03:07:10.718863 23247 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:07:10.718963 23189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b13b5707bade43958f578a3e7d1bc241"
I20260924 03:07:10.719067 23189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b13b5707bade43958f578a3e7d1bc241 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:07:10.719987 23189 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b13b5707bade43958f578a3e7d1bc241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e459bd18b0084b9dbe2b19d6344eb22d in term 1.
I20260924 03:07:10.720194 23015 leader_election.cc:302] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b13b5707bade43958f578a3e7d1bc241, e459bd18b0084b9dbe2b19d6344eb22d; no voters:
I20260924 03:07:10.720350 23222 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:07:10.720508 23222 raft_consensus.cc:686] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d [term 1 LEADER]: Becoming Leader. State: Replica: e459bd18b0084b9dbe2b19d6344eb22d, State: Running, Role: LEADER
I20260924 03:07:10.720604 23222 consensus_queue.cc:237] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35663 } } peers { permanent_uuid: "9a6aa4eccbf14acd842659653297f685" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46079 } } peers { permanent_uuid: "b13b5707bade43958f578a3e7d1bc241" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43349 } }
I20260924 03:07:10.720993 23222 sys_catalog.cc:455] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e459bd18b0084b9dbe2b19d6344eb22d. Latest consensus state: current_term: 1 leader_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35663 } } peers { permanent_uuid: "9a6aa4eccbf14acd842659653297f685" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46079 } } peers { permanent_uuid: "b13b5707bade43958f578a3e7d1bc241" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43349 } } }
I20260924 03:07:10.721071 23222 sys_catalog.cc:458] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d [sys.catalog]: This master's current role is: LEADER
I20260924 03:07:10.721812 23147 master_runner.cc:186] Error getting master registration for 127.0.98.61:46079: OK, instance_id { permanent_uuid: "9a6aa4eccbf14acd842659653297f685" instance_seqno: 1790219230496576 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:07:10.724318 23147 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:07:10.727813 23255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:07:10.728075 23255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:07:10.730050 23189 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b13b5707bade43958f578a3e7d1bc241 [term 1 FOLLOWER]: Refusing update from remote peer e459bd18b0084b9dbe2b19d6344eb22d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:07:10.730391 23253 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b13b5707bade43958f578a3e7d1bc241" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:07:10.731813 23236 sys_catalog.cc:455] T 00000000000000000000000000000000 P b13b5707bade43958f578a3e7d1bc241 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e459bd18b0084b9dbe2b19d6344eb22d. Latest consensus state: current_term: 1 leader_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35663 } } peers { permanent_uuid: "9a6aa4eccbf14acd842659653297f685" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46079 } } peers { permanent_uuid: "b13b5707bade43958f578a3e7d1bc241" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43349 } } }
I20260924 03:07:10.731905 23236 sys_catalog.cc:458] T 00000000000000000000000000000000 P b13b5707bade43958f578a3e7d1bc241 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:07:10.738162 23140 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43349 } attempt: 3
I20260924 03:07:10.741580 23222 sys_catalog.cc:455] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35663 } } peers { permanent_uuid: "9a6aa4eccbf14acd842659653297f685" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46079 } } peers { permanent_uuid: "b13b5707bade43958f578a3e7d1bc241" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43349 } } }
I20260924 03:07:10.741698 23222 sys_catalog.cc:458] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d [sys.catalog]: This master's current role is: LEADER
I20260924 03:07:10.741950 23236 sys_catalog.cc:455] T 00000000000000000000000000000000 P b13b5707bade43958f578a3e7d1bc241 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35663 } } peers { permanent_uuid: "9a6aa4eccbf14acd842659653297f685" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46079 } } peers { permanent_uuid: "b13b5707bade43958f578a3e7d1bc241" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43349 } } }
I20260924 03:07:10.742056 23236 sys_catalog.cc:458] T 00000000000000000000000000000000 P b13b5707bade43958f578a3e7d1bc241 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:07:10.742209 23257 mvcc.cc:205] Tried to move back new op lower bound from 7332737969066938368 to 7332737969032232960. Current Snapshot: MvccSnapshot[applied={T|T < 7332737969066938368}]
Sep 24 03:07:10 dist-test-slave-wgsb krb5kdc[22984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 03:07:10.746228 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.738413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43349 (local address 127.0.0.1:45612)
0924 03:07:10.738502 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0924 03:07:10.738513 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:07:10.738553 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:07:10.738791 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:07:10.738792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:07:10.738961 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:07:10.739001 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:10.739003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.739957 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.739958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:10.740752 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:07:10.740754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.741274 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.741275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:07:10.741344 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:10.741886 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:07:10.741904 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:07:10.743301 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:07:10.745105 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:10.745106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:10.745107 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:10.745309 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:10.745531 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:07:10.745532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:07:10.745532 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:07:10.745589 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:07:10.745879 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:07:10.745880 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:07:10.745922 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 03:07:10.746009 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 03:07:10.746109 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 03:07:10.746572 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.738623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45612 (local address 127.0.98.60:43349)
0924 03:07:10.738665 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:10.738667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:10.738673 (+ 6us) server_negotiation.cc:407] Connection header received
0924 03:07:10.738693 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:10.738694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:10.738705 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:10.738761 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:10.739083 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.739862 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.740841 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.741175 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.741256 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:10.743379 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:10.743381 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:10.743381 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:10.743389 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:10.745010 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.745406 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.745407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.745408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.745479 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.745662 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.745663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.745664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.745792 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:10.745853 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:10.746407 (+ 554us) server_negotiation.cc:299] Negotiation successful
0924 03:07:10.746490 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 03:07:10.748067 23140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a6aa4eccbf14acd842659653297f685: Bootstrap starting.
I20260924 03:07:10.748793 23140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a6aa4eccbf14acd842659653297f685: Neither blocks nor log segments found. Creating new log.
I20260924 03:07:10.749097 23140 log.cc:824] T 00000000000000000000000000000000 P 9a6aa4eccbf14acd842659653297f685: Log is configured to *not* fsync() on all Append() calls
I20260924 03:07:10.749490 23255 catalog_manager.cc:1347] Generated new cluster ID: ab013334798248b2a1ae3a015eb5e758
I20260924 03:07:10.749540 23255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:07:10.750248 23140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a6aa4eccbf14acd842659653297f685: No bootstrap required, opened a new log
I20260924 03:07:10.753160 23140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a6aa4eccbf14acd842659653297f685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35663 } } peers { permanent_uuid: "9a6aa4eccbf14acd842659653297f685" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46079 } } peers { permanent_uuid: "b13b5707bade43958f578a3e7d1bc241" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43349 } }
I20260924 03:07:10.753520 23140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a6aa4eccbf14acd842659653297f685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a6aa4eccbf14acd842659653297f685, State: Initialized, Role: FOLLOWER
I20260924 03:07:10.753733 23140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a6aa4eccbf14acd842659653297f685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35663 } } peers { permanent_uuid: "9a6aa4eccbf14acd842659653297f685" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46079 } } peers { permanent_uuid: "b13b5707bade43958f578a3e7d1bc241" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43349 } }
I20260924 03:07:10.754484 23113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a6aa4eccbf14acd842659653297f685 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:07:10.755406 23113 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9a6aa4eccbf14acd842659653297f685 [term 1 FOLLOWER]: Refusing update from remote peer e459bd18b0084b9dbe2b19d6344eb22d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:07:10.755811 23263 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a6aa4eccbf14acd842659653297f685" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:07:10.756189 23114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9a6aa4eccbf14acd842659653297f685"
I20260924 03:07:10.756410 23260 mvcc.cc:205] Tried to move back new op lower bound from 7332737969066938368 to 7332737969032232960. Current Snapshot: MvccSnapshot[applied={T|T < 7332737969066938368}]
I20260924 03:07:10.756953 23114 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9a6aa4eccbf14acd842659653297f685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e459bd18b0084b9dbe2b19d6344eb22d in term 1.
I20260924 03:07:10.754398 23140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a6aa4eccbf14acd842659653297f685 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:07:10.761680 23144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.722027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50014 (local address 127.0.98.61:46079)
0924 03:07:10.722068 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:10.722071 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:10.731571 (+ 9500us) server_negotiation.cc:407] Connection header received
0924 03:07:10.731708 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:10.731709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:10.731721 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:10.731752 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:10.732720 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.733470 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.734585 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.734904 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.734969 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:10.737852 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:10.737854 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:10.737854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:10.737862 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:10.739653 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.742150 (+ 2497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.742151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.742151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.742216 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.744416 (+ 2200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.744418 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.744418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.744564 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:10.744647 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:10.761487 (+ 16840us) server_negotiation.cc:299] Negotiation successful
0924 03:07:10.761588 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 03:07:10.762952 23281 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a6aa4eccbf14acd842659653297f685: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:07:10.763059 23281 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:07:10.763236 23270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a6aa4eccbf14acd842659653297f685 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35663 } } peers { permanent_uuid: "9a6aa4eccbf14acd842659653297f685" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46079 } } peers { permanent_uuid: "b13b5707bade43958f578a3e7d1bc241" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43349 } } }
I20260924 03:07:10.763356 23270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a6aa4eccbf14acd842659653297f685 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:07:10.763439 23270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a6aa4eccbf14acd842659653297f685 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e459bd18b0084b9dbe2b19d6344eb22d. Latest consensus state: current_term: 1 leader_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35663 } } peers { permanent_uuid: "9a6aa4eccbf14acd842659653297f685" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46079 } } peers { permanent_uuid: "b13b5707bade43958f578a3e7d1bc241" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43349 } } }
I20260924 03:07:10.763532 23270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a6aa4eccbf14acd842659653297f685 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:07:10.763620 23270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a6aa4eccbf14acd842659653297f685 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35663 } } peers { permanent_uuid: "9a6aa4eccbf14acd842659653297f685" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46079 } } peers { permanent_uuid: "b13b5707bade43958f578a3e7d1bc241" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43349 } } }
I20260924 03:07:10.763729 23270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a6aa4eccbf14acd842659653297f685 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:07:10.764981 23268 mvcc.cc:205] Tried to move back new op lower bound from 7332737969066938368 to 7332737969032232960. Current Snapshot: MvccSnapshot[applied={T|T < 7332737969066938368}]
I20260924 03:07:10.766700 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.729763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50836 (local address 127.0.98.62:35663)
0924 03:07:10.729909 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:10.729911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:10.732803 (+ 2892us) server_negotiation.cc:407] Connection header received
0924 03:07:10.732897 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:10.732898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:10.732908 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:10.732937 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:10.733635 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.734348 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.735759 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.736082 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.736147 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:10.763711 (+ 27564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:10.763712 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:10.763713 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:10.763721 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:10.765343 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.765755 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.765756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.765756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.765809 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.766026 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.766027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.766027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.766154 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:10.766237 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:10.766519 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 03:07:10.766601 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260924 03:07:10.770206 23220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:10.731120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45610 (local address 127.0.98.60:43349)
0924 03:07:10.731156 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:10.731158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:10.734669 (+ 3511us) server_negotiation.cc:407] Connection header received
0924 03:07:10.734760 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:10.734761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:10.734771 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:10.734799 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:10.737825 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.738555 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.755737 (+ 17182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:10.756075 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:10.756125 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:10.767324 (+ 11199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:07:10.767326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:07:10.767326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:07:10.767334 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:07:10.769006 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.769421 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.769422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.769423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.769475 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:07:10.769653 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:07:10.769654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:07:10.769654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:07:10.769777 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:07:10.769842 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:10.770033 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 03:07:10.770113 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:07:10.771647 23253 sys_catalog.cc:455] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e459bd18b0084b9dbe2b19d6344eb22d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35663 } } peers { permanent_uuid: "9a6aa4eccbf14acd842659653297f685" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46079 } } peers { permanent_uuid: "b13b5707bade43958f578a3e7d1bc241" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43349 } } }
I20260924 03:07:10.771760 23253 sys_catalog.cc:458] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d [sys.catalog]: This master's current role is: LEADER
I20260924 03:07:10.773102 23255 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:07:10.773840 23255 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:07:10.778626 23255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e459bd18b0084b9dbe2b19d6344eb22d: Generated new TSK 0
I20260924 03:07:10.778791 23255 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 03:07:10.796294 23068 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:07:10.795006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50842 (local address 127.0.98.62:35663)
0924 03:07:10.795051 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:10.795053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:10.795648 (+ 595us) server_negotiation.cc:407] Connection header received
0924 03:07:10.795755 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:10.795756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:10.795768 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:10.795799 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:10.796228 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50842: BlockingRecv error: recv got EOF from 127.0.0.1:50842 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 03:07:10.797250 23220 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:07:10.795346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45628 (local address 127.0.98.60:43349)
0924 03:07:10.795398 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:10.795400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:10.796730 (+ 1330us) server_negotiation.cc:407] Connection header received
0924 03:07:10.796860 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:10.796861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:10.796872 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:10.796902 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:10.797191 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45628: BlockingRecv error: recv got EOF from 127.0.0.1:45628 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 03:07:10.798028 23144 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:07:10.796617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50024 (local address 127.0.98.61:46079)
0924 03:07:10.796656 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:10.796658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:10.797445 (+ 787us) server_negotiation.cc:407] Connection header received
0924 03:07:10.797601 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:10.797602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:10.797612 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:10.797641 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:10.797961 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50024: BlockingRecv error: recv got EOF from 127.0.0.1:50024 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 03:07:10.800231 23144 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:07:10.799533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50040 (local address 127.0.98.61:46079)
0924 03:07:10.799566 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:10.799568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:10.799748 (+ 180us) server_negotiation.cc:407] Connection header received
0924 03:07:10.799850 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:10.799851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:10.799868 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:10.799896 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:10.800186 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50040: BlockingRecv error: recv got EOF from 127.0.0.1:50040 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 03:07:10.803249 23144 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:07:10.802535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50052 (local address 127.0.98.61:46079)
0924 03:07:10.802567 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:10.802569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:10.802776 (+ 207us) server_negotiation.cc:407] Connection header received
0924 03:07:10.802859 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:10.802860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:10.802869 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:10.802897 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:10.803206 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50052: BlockingRecv error: recv got EOF from 127.0.0.1:50052 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 03:07:10.807179 23144 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:07:10.806433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50066 (local address 127.0.98.61:46079)
0924 03:07:10.806468 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:10.806471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:10.806678 (+ 207us) server_negotiation.cc:407] Connection header received
0924 03:07:10.806773 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:10.806774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:10.806785 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:10.806812 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:10.807130 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50066: BlockingRecv error: recv got EOF from 127.0.0.1:50066 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 03:07:10.813148 23144 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:07:10.812390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50068 (local address 127.0.98.61:46079)
0924 03:07:10.812426 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:10.812428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:10.812650 (+ 222us) server_negotiation.cc:407] Connection header received
0924 03:07:10.812743 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:10.812744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:10.812754 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:10.812783 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:10.813096 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50068: BlockingRecv error: recv got EOF from 127.0.0.1:50068 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 03:07:10.823256 23144 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:07:10.822360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50080 (local address 127.0.98.61:46079)
0924 03:07:10.822423 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:10.822425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:10.822653 (+ 228us) server_negotiation.cc:407] Connection header received
0924 03:07:10.822774 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:10.822775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:10.822786 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:10.822816 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:10.823187 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50080: BlockingRecv error: recv got EOF from 127.0.0.1:50080 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 03:07:10.841558 23144 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:07:10.840639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50096 (local address 127.0.98.61:46079)
0924 03:07:10.840684 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:10.840687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:10.840935 (+ 248us) server_negotiation.cc:407] Connection header received
0924 03:07:10.841042 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:10.841043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:10.841054 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:10.841085 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:10.841497 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50096: BlockingRecv error: recv got EOF from 127.0.0.1:50096 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 03:07:10.875995 23144 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:07:10.875002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50098 (local address 127.0.98.61:46079)
0924 03:07:10.875047 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:10.875049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:10.875285 (+ 236us) server_negotiation.cc:407] Connection header received
0924 03:07:10.875438 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:10.875440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:10.875452 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:10.875482 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:10.875934 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50098: BlockingRecv error: recv got EOF from 127.0.0.1:50098 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 03:07:10.942122 23144 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:07:10.941389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50104 (local address 127.0.98.61:46079)
0924 03:07:10.941455 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:10.941457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:10.941513 (+ 56us) server_negotiation.cc:407] Connection header received
0924 03:07:10.941572 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:10.941574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:10.941585 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:10.941669 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:10.942057 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50104: BlockingRecv error: recv got EOF from 127.0.0.1:50104 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 03:07:11.072101 23144 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:07:11.071404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50118 (local address 127.0.98.61:46079)
0924 03:07:11.071472 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:11.071474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:11.071506 (+ 32us) server_negotiation.cc:407] Connection header received
0924 03:07:11.071579 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:11.071580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:11.071634 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:11.071702 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:11.072054 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50118: BlockingRecv error: recv got EOF from 127.0.0.1:50118 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 03:07:11.332854 23144 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:07:11.330515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60064 (local address 127.0.98.61:46079)
0924 03:07:11.330791 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:11.330794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:11.330817 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:07:11.331813 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:11.331816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:11.331857 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:11.331946 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:07:11.332707 (+ 761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60064: BlockingRecv error: recv got EOF from 127.0.0.1:60064 (error 108)
Metrics: {"server-negotiator.queue_time_us":232}
I20260924 03:07:11.719341 23247 catalog_manager.cc:1259] Loaded cluster ID: ab013334798248b2a1ae3a015eb5e758
I20260924 03:07:11.719450 23247 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b13b5707bade43958f578a3e7d1bc241: loading cluster ID for follower catalog manager: success
I20260924 03:07:11.720253 23247 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b13b5707bade43958f578a3e7d1bc241: acquiring CA information for follower catalog manager: success
I20260924 03:07:11.720582 23247 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b13b5707bade43958f578a3e7d1bc241: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:07:11.763574 23281 catalog_manager.cc:1259] Loaded cluster ID: ab013334798248b2a1ae3a015eb5e758
I20260924 03:07:11.763674 23281 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9a6aa4eccbf14acd842659653297f685: loading cluster ID for follower catalog manager: success
I20260924 03:07:11.764879 23281 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9a6aa4eccbf14acd842659653297f685: acquiring CA information for follower catalog manager: success
I20260924 03:07:11.765252 23281 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9a6aa4eccbf14acd842659653297f685: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:07:11.850739 23347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:11.846881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60072 (local address 127.0.98.61:46079)
0924 03:07:11.846986 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:11.846988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:11.847811 (+ 823us) server_negotiation.cc:407] Connection header received
0924 03:07:11.847852 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:11.847853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:11.847920 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:11.848027 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:07:11.848300 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:11.848844 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:11.849910 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:11.850113 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:11.850187 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:11.850333 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:07:11.850481 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:07:11.850546 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:11.850589 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 03:07:11.850609 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":47,"threads_started":1}
I20260924 03:07:11.855523 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:11.852797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57574 (local address 127.0.98.60:43349)
0924 03:07:11.852933 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:11.852935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:11.852985 (+ 50us) server_negotiation.cc:407] Connection header received
0924 03:07:11.853038 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:11.853039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:11.853073 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:11.853152 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:07:11.853299 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:11.853777 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:11.854576 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:11.854770 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:11.854866 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:11.855071 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:07:11.855195 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:07:11.855271 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:11.855421 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 03:07:11.855442 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
I20260924 03:07:11.860384 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:11.857860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57582 (local address 127.0.98.60:43349)
0924 03:07:11.857900 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:11.857901 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:11.858037 (+ 136us) server_negotiation.cc:407] Connection header received
0924 03:07:11.858096 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:11.858097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:11.858108 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:11.858161 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:07:11.858292 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:11.858806 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:11.859625 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:11.859854 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:11.859914 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:11.860097 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:07:11.860128 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:07:11.860155 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:11.860248 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 03:07:11.860278 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:07:11.860460 23347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:11.857467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60080 (local address 127.0.98.61:46079)
0924 03:07:11.857532 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:11.857534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:11.857544 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:07:11.857574 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:11.857575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:11.857586 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:11.857664 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:07:11.857791 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:11.858531 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:11.859648 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:11.859946 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:11.859985 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:11.860187 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:07:11.860218 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:07:11.860268 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:11.860362 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 03:07:11.860384 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 03:07:11.861240 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:07:11.857464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52866 (local address 127.0.98.62:35663)
0924 03:07:11.857560 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 03:07:11.857562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:07:11.857726 (+ 164us) server_negotiation.cc:407] Connection header received
0924 03:07:11.857858 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:07:11.857859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:07:11.857928 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:07:11.858011 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:07:11.858187 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:11.858947 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:11.859985 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:07:11.860317 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:07:11.860380 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:07:11.860849 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:07:11.860967 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:07:11.861028 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 03:07:11.861106 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 03:07:11.861135 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":52,"threads_started":1}
I20260924 03:07:11.862962 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 22997
I20260924 03:07:11.868494 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 23072
I20260924 03:07:11.873497 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskptPCbv/build/release/bin/kudu with pid 23147
2026-09-24T03:07:11Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2856 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5772 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (283593 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST